Oct 30 23:18:35.946750 ip-10-0-137-81 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:18:36.279969 ip-10-0-137-81 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:18:36.279969 ip-10-0-137-81 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:18:36.279969 ip-10-0-137-81 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:18:36.279969 ip-10-0-137-81 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:18:36.284316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284233 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:18:36.284316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284318 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284323 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284327 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284330 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284334 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284336 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284339 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284342 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284345 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284347 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284350 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284352 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284355 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284358 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284360 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284362 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284365 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284368 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284371 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:18:36.284383 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284373 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284376 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284378 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284380 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284391 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284402 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284405 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284407 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284410 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284413 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284415 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284418 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284420 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284423 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284426 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284428 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284431 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284433 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284435 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284438 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:18:36.284837 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284440 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284443 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284445 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284447 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284450 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284452 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284455 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284478 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284483 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284486 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284489 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284492 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284494 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284497 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284501 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284504 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284506 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284509 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284512 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:18:36.285316 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284514 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284517 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284520 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284522 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284525 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284527 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284530 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284532 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284535 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284537 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284540 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284542 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284545 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284547 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284550 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284552 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284555 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284557 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284559 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284563 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:18:36.285770 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284565 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284568 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284571 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284574 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284576 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284579 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284581 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284585 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284588 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284590 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284593 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284595 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284599 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284602 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284604 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284607 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284609 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284611 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284614 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284617 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:18:36.286241 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284619 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284621 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284624 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284626 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284629 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284631 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.284633 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285482 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285490 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285493 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285496 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285499 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285502 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285505 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285508 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285511 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285513 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285516 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285521 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:18:36.286722 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285525 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285529 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285531 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285534 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285536 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285539 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285542 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285544 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285547 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285550 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285552 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285555 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285557 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285560 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285562 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285565 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285567 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285570 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285572 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:18:36.287173 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285575 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285578 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285580 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285584 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285588 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285592 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285595 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285597 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285600 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285603 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285605 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285608 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285611 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285613 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285616 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285619 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285621 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285624 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285626 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:18:36.287634 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285628 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285631 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285633 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285636 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285638 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285641 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285644 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285646 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285649 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285651 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285654 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285657 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285659 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285662 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285665 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285667 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285669 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285672 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285674 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:18:36.288087 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285677 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285680 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285683 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285685 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285688 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285691 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285693 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285696 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285698 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285701 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285703 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285706 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285708 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285711 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285713 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285716 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285718 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285720 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285723 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285725 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:18:36.288558 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285728 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285731 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285733 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285737 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285739 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285741 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285744 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285746 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285749 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285751 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285754 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285756 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285758 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285761 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285765 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285767 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.285770 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286275 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286280 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286284 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:18:36.289056 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286287 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286290 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286293 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286295 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286298 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286300 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286302 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286305 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286307 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286310 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286312 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286315 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286317 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286320 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286324 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286327 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286330 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286333 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286336 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286338 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:18:36.289542 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286341 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286343 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286346 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286348 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286351 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286353 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286356 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286358 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286361 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286364 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286366 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286370 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286373 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286375 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286378 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286380 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286383 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286385 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286388 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:18:36.290016 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286390 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286393 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286395 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286397 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286400 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286402 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286405 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286408 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286411 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286413 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286416 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286418 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286421 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286423 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286426 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286428 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286430 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286433 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286435 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286438 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:18:36.290516 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286440 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286442 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286445 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286448 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286451 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286454 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286456 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286472 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286475 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286478 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286480 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286483 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286485 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286488 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286490 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286493 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286496 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286498 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286500 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:18:36.290978 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286503 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286506 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286509 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286511 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286514 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286516 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286519 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286521 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286524 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286526 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286528 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286531 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286533 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286536 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286538 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286541 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286543 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286549 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286552 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:18:36.291420 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286554 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286557 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286559 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286562 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286567 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.286570 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287340 2581 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287353 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287361 2581 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287366 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287370 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287373 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287378 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287382 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287387 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287391 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287394 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287397 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287400 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287403 2581 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287406 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287409 2581 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:18:36.291911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287411 2581 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287414 2581 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287418 2581 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287420 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287423 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287428 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287432 2581 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287436 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287438 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287442 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287445 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287448 2581 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287451 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287454 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287470 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287474 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287479 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287481 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287485 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287488 2581 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287491 2581 flags.go:64] FLAG: --enable-server="true" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287494 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287499 2581 flags.go:64] FLAG: --event-burst="100" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287502 2581 flags.go:64] FLAG: --event-qps="50" Oct 30 23:18:36.292424 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287507 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287510 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287512 2581 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287517 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287520 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287523 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287526 2581 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287529 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287532 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287535 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287538 2581 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287541 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287544 2581 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287548 2581 flags.go:64] FLAG: --feature-gates="" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287552 2581 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287555 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287558 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287561 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287565 2581 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287568 2581 flags.go:64] FLAG: --help="false" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287571 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287575 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287578 2581 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:18:36.293035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287580 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287584 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287587 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287590 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287593 2581 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287595 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287598 2581 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287601 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287604 2581 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287608 2581 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287611 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287614 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287617 2581 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287620 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287623 2581 flags.go:64] FLAG: --lock-file="" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287625 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287628 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287631 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287636 2581 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287639 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287643 2581 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287646 2581 flags.go:64] FLAG: --logging-format="text" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287649 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287653 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:18:36.293609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287655 2581 flags.go:64] FLAG: --manifest-url="" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287675 2581 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287681 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287684 2581 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287689 2581 flags.go:64] FLAG: --max-pods="110" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287692 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287695 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287698 2581 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287701 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287703 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287706 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287709 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287716 2581 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287719 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287722 2581 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287725 2581 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287728 2581 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287731 2581 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287734 2581 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287737 2581 flags.go:64] FLAG: --port="10250" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287740 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287743 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d02ee9bc8e5b0505" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287746 2581 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287749 2581 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287752 2581 flags.go:64] FLAG: --register-node="true" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287754 2581 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:18:36.294202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287758 2581 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287761 2581 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287764 2581 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287766 2581 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287771 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287774 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287777 2581 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287780 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287783 2581 flags.go:64] FLAG: --runonce="false" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287785 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287789 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287793 2581 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287796 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287798 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287801 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287804 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287807 2581 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287810 2581 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287813 2581 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287816 2581 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287819 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287821 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287824 2581 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287827 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287833 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:18:36.294827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287836 2581 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287838 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287842 2581 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287844 2581 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287847 2581 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287850 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287853 2581 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287856 2581 flags.go:64] FLAG: --v="2" Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287860 2581 flags.go:64] FLAG: --version="false" Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287864 2581 flags.go:64] FLAG: --vmodule="" Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287869 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.287874 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288045 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288048 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288051 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288055 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288057 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288060 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288063 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288071 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288073 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288076 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288079 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:18:36.295572 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288081 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288083 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288086 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288089 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288092 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288094 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288097 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288100 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288103 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288105 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288108 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288110 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288113 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288116 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288118 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288121 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288124 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288126 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288130 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288134 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:18:36.296119 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288139 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288141 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288144 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288146 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288149 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288152 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288154 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288157 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288159 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288167 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288170 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288172 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288175 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288177 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288180 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288182 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288185 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288187 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288190 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:18:36.296600 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288192 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288201 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288204 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288206 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288208 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288211 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288213 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288216 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288218 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288221 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288223 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288225 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288228 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288232 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288234 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288237 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288239 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288242 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288244 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:18:36.297071 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288247 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288249 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288252 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288254 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288262 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288266 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288269 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288271 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288274 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288276 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288279 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288281 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288284 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288287 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288289 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288292 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288294 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288297 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288299 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288302 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:18:36.297536 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288304 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288307 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288309 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288311 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288314 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288316 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288320 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288322 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288325 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288328 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288331 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288333 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288337 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288341 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288343 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288346 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:18:36.298018 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288349 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.288382 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288433 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288437 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288440 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288443 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288445 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288448 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288450 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288453 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288455 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288471 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288474 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288476 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288479 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288481 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:18:36.298398 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288484 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288486 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288489 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288492 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288494 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288497 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288499 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288503 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288506 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288523 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288528 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288531 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288534 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288537 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288539 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288542 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288545 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288548 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288556 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288560 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:18:36.298790 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288563 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288565 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288568 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288572 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288576 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288579 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288582 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288584 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288587 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288589 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288592 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288595 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288597 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288600 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288602 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288605 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288607 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288610 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288612 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:18:36.299268 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288616 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288619 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288621 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288623 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288626 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288628 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288631 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288633 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288636 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288638 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288642 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288645 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288653 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288655 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288658 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288660 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288663 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288665 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288669 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288672 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:18:36.299718 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288676 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288678 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288682 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288684 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288687 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288689 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288691 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288694 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288697 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288699 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288701 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288704 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288707 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288710 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288712 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288718 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288721 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288723 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288726 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:18:36.300179 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288728 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288731 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288733 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288737 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288740 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288742 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288746 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288749 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288752 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288754 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288757 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288759 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288762 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.288765 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.296123 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.296138 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.296748 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.296755 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296827 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:18:36.300774 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296833 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296837 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296840 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296843 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296846 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296856 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296859 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296862 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296864 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296867 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296870 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296872 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296875 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296877 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296880 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296882 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296885 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296894 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296897 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296907 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:18:36.301269 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296910 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296914 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296918 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296922 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296926 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296929 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296932 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296934 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296937 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296940 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296942 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296945 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296947 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296950 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296952 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296954 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296957 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296959 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296961 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:18:36.301785 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296964 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296966 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296969 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296971 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296974 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296976 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296978 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296981 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296983 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296986 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296988 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296990 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.296993 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297001 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297004 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297007 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297009 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297012 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297024 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:18:36.302348 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297028 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297032 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297035 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297037 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297040 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297042 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297045 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297047 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297050 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297053 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297055 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297058 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297060 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297063 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297065 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297067 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297070 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297072 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297075 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297078 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:18:36.302852 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297080 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297083 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297085 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297088 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297091 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297093 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297096 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297104 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297107 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297110 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297112 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297115 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297117 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297120 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297122 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297125 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297127 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297130 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297133 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297135 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:18:36.303360 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297137 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297140 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297142 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297144 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297147 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297149 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297151 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.297180 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297370 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297375 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297378 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297380 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297383 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297386 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297388 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:18:36.304063 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297391 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297393 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297396 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297399 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297401 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297410 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297412 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297415 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297418 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297420 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297423 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297425 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297428 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297430 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297433 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297435 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297437 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297440 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297443 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297445 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:18:36.304453 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297448 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297450 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297453 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297455 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297474 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297478 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297482 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297486 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297489 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297493 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297497 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297500 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297503 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297505 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297508 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297510 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297513 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297515 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297525 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:18:36.304980 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297528 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297531 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297534 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297536 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297539 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297541 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297544 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297547 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297549 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297552 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297554 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297556 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297559 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297561 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297564 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297568 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297571 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297574 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297576 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297579 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:18:36.305867 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297581 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297584 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297587 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297589 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297592 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297594 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297596 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297598 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297601 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297603 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297606 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297608 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297616 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297619 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297622 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297624 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297626 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297629 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297631 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297634 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:18:36.306345 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297636 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297639 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297641 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297643 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297646 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297648 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297650 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297653 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297655 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297658 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297660 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297662 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297665 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297668 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297670 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297672 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297675 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297677 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297679 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:18:36.306828 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297682 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.297706 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297763 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297768 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297773 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297777 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297785 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297788 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297791 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297794 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297797 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297799 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297801 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297804 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297806 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297809 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:18:36.307278 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297811 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297814 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297816 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297819 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297821 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297824 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297826 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297828 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297831 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297834 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297836 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297839 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297841 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297844 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297852 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297855 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297857 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297860 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297862 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:18:36.307742 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297865 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297869 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297871 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297874 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297882 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297885 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297887 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297890 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297892 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297895 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297898 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297900 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297902 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297905 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297909 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297911 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297914 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297916 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297919 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297921 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:18:36.308233 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297924 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297927 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297930 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297932 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297935 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297937 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297939 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297942 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297944 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297947 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297949 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297952 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297954 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297958 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297961 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297964 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297966 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297976 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297978 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:18:36.308731 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297981 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297985 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297989 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297991 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297994 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297996 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.297999 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298002 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298005 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298008 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298010 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298012 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298015 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298017 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298020 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298022 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298025 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298028 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298030 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:18:36.309197 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298033 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298035 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298038 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298040 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298042 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298045 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298047 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298053 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298057 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298060 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298063 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298065 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298073 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298076 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:36.298079 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.298134 2581 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.300809 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.301681 2581 server.go:1014] "Starting client certificate rotation" Oct 30 23:18:36.309698 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.301808 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:18:36.310107 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.301870 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:18:36.310107 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.305851 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:18:36.310107 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.307285 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:18:36.315248 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.315231 2581 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:18:36.318564 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.318546 2581 log.go:25] "Validated CRI v1 image API" Oct 30 23:18:36.319360 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.319337 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:18:36.321655 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.321637 2581 fs.go:135] Filesystem UUIDs: map[0d2073c4-0a4a-4a34-b277-998c64e55246:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a4c4fc0d-419f-4a9e-9c2d-7272f9470270:/dev/nvme0n1p3] Oct 30 23:18:36.321708 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.321655 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 23:18:36.326980 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.326858 2581 manager.go:217] Machine: {Timestamp:2026-10-30 23:18:36.325692226 +0000 UTC m=+0.278799544 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095395 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22e1783cebdd2979b4726d824250c5 SystemUUID:ec22e178-3ceb-dd29-79b4-726d824250c5 BootID:a297dd64-45b4-4e7d-8665-04afd2be3a4b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:56:c0:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:56:c0:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:27:a2:b9:cd:c8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:18:36.327051 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.326984 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:18:36.327110 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.327099 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:18:36.329597 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.329576 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:18:36.329744 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.329600 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-81.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:18:36.329788 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.329747 2581 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:18:36.329788 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.329755 2581 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:18:36.329788 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.329787 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:18:36.329862 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.329806 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:18:36.330628 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.330618 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:18:36.334431 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.334419 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:18:36.334585 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.334575 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:18:36.335132 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.335121 2581 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:18:36.335164 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.335139 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:18:36.335164 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.335149 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:18:36.335164 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.335158 2581 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:18:36.335251 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.335166 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:18:36.335627 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.335604 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:18:36.337515 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.337498 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:18:36.345014 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.344418 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:18:36.346204 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346189 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:18:36.346247 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346224 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:18:36.346395 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346385 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:18:36.346426 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346402 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:18:36.346426 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346408 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:18:36.346426 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346413 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:18:36.346426 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346419 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:18:36.346555 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346426 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:18:36.346881 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346872 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:18:36.346911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346882 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:18:36.346911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346890 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:18:36.346911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346896 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:18:36.346911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346904 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:18:36.346911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.346912 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:18:36.347551 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.347540 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:18:36.347582 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.347554 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:18:36.351656 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.351644 2581 server.go:1263] "Started kubelet" Oct 30 23:18:36.351768 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.351737 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:18:36.351815 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.351736 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:18:36.351857 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.351823 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:18:36.352560 ip-10-0-137-81 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:18:36.353103 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.352978 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-81.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 23:18:36.353179 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.353131 2581 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:18:36.353730 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.353649 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:18:36.357878 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.357851 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 23:18:36.357955 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.357898 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 23:18:36.358049 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.358033 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-81.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 23:18:36.358376 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.357616 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-81.us-east-2.compute.internal.18e371e7a58037fc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-81.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:18:36.35161702 +0000 UTC m=+0.304724338,LastTimestamp:2026-10-30 23:18:36.35161702 +0000 UTC m=+0.304724338,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-81.us-east-2.compute.internal,}" Oct 30 23:18:36.358514 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.358494 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:18:36.359715 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.359698 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:18:36.359859 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.359836 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:36.360492 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.360477 2581 factory.go:55] Registering systemd factory Oct 30 23:18:36.360492 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.360494 2581 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:18:36.360620 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.360534 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:18:36.360620 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.360577 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:18:36.360620 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.360609 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:18:36.360755 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.360708 2581 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:18:36.360755 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.360719 2581 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:18:36.360927 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.360915 2581 factory.go:153] Registering CRI-O factory Oct 30 23:18:36.360984 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.360931 2581 factory.go:223] Registration of the crio container factory successfully Oct 30 23:18:36.360984 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.360976 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:18:36.361081 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.360995 2581 factory.go:103] Registering Raw factory Oct 30 23:18:36.361081 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.361027 2581 manager.go:1199] Started watching for new ooms in manager Oct 30 23:18:36.361963 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.361948 2581 manager.go:319] Starting recovery of all containers Oct 30 23:18:36.369861 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.369829 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-81.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 23:18:36.375054 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.375036 2581 manager.go:324] Recovery completed Oct 30 23:18:36.376195 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.376166 2581 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 23:18:36.377795 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.377775 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 23:18:36.379432 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.379420 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:18:36.381177 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.381162 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:18:36.381226 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.381192 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:18:36.381226 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.381204 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:18:36.381748 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.381732 2581 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:18:36.381748 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.381743 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:18:36.381885 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.381764 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:18:36.381885 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.381768 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:18:36.382391 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.382375 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:18:36.382450 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.382406 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:18:36.382450 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.382416 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:18:36.383996 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.383984 2581 policy_none.go:50] "Start" Oct 30 23:18:36.384057 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.384000 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:18:36.384057 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.384010 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:18:36.385176 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.385111 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-81.us-east-2.compute.internal.18e371e7a7434be3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-81.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-81.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:18:36.381178851 +0000 UTC m=+0.334286171,LastTimestamp:2026-10-30 23:18:36.381178851 +0000 UTC m=+0.334286171,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-81.us-east-2.compute.internal,}" Oct 30 23:18:36.386006 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.385995 2581 policy_none.go:44] "Start" Oct 30 23:18:36.397784 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.397699 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-81.us-east-2.compute.internal.18e371e7a7439392 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-81.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-81.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:18:36.381197202 +0000 UTC m=+0.334304517,LastTimestamp:2026-10-30 23:18:36.381197202 +0000 UTC m=+0.334304517,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-81.us-east-2.compute.internal,}" Oct 30 23:18:36.400410 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.400348 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-81.us-east-2.compute.internal.18e371e7a743be2f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-81.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-81.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:18:36.381208111 +0000 UTC m=+0.334315427,LastTimestamp:2026-10-30 23:18:36.381208111 +0000 UTC m=+0.334315427,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-81.us-east-2.compute.internal,}" Oct 30 23:18:36.401953 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.401936 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:18:36.411737 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.411664 2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-81.us-east-2.compute.internal.18e371e7a7434be3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-81.us-east-2.compute.internal.18e371e7a7434be3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-81.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-81.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:18:36.381178851 +0000 UTC m=+0.334286171,LastTimestamp:2026-10-30 23:18:36.382391259 +0000 UTC m=+0.335498589,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-81.us-east-2.compute.internal,}" Oct 30 23:18:36.415722 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.415648 2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-81.us-east-2.compute.internal.18e371e7a7439392\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-81.us-east-2.compute.internal.18e371e7a7439392 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-81.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-81.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:18:36.381197202 +0000 UTC m=+0.334304517,LastTimestamp:2026-10-30 23:18:36.382411021 +0000 UTC m=+0.335518338,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-81.us-east-2.compute.internal,}" Oct 30 23:18:36.426441 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.426365 2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-81.us-east-2.compute.internal.18e371e7a743be2f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-81.us-east-2.compute.internal.18e371e7a743be2f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-81.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-81.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:18:36.381208111 +0000 UTC m=+0.334315427,LastTimestamp:2026-10-30 23:18:36.382420041 +0000 UTC m=+0.335527357,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-81.us-east-2.compute.internal,}" Oct 30 23:18:36.426665 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.426649 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mc6pp" Oct 30 23:18:36.429256 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.429230 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:18:36.432428 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.432416 2581 manager.go:341] "Starting Device Plugin manager" Oct 30 23:18:36.432478 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.432452 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:18:36.432478 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.432476 2581 server.go:87] "Starting device plugin registration server" Oct 30 23:18:36.432816 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.432802 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:18:36.432878 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.432817 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:18:36.432946 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.432926 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:18:36.433286 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.433029 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:18:36.433286 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.433041 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:18:36.433586 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.433556 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:18:36.433653 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.433627 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:36.434807 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.434738 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-81.us-east-2.compute.internal.18e371e7aa509e9b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated limits on system reserved cgroup /system.slice,Source:EventSource{Component:kubelet,Host:ip-10-0-137-81.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:18:36.432383643 +0000 UTC m=+0.385490947,LastTimestamp:2026-10-30 23:18:36.432383643 +0000 UTC m=+0.385490947,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-81.us-east-2.compute.internal,}" Oct 30 23:18:36.439400 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.439383 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mc6pp" Oct 30 23:18:36.525902 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.525741 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:18:36.527173 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.527159 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:18:36.527250 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.527179 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:18:36.527250 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.527200 2581 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:18:36.527250 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.527241 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 23:18:36.530982 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.530941 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:18:36.533188 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.533174 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:18:36.534001 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.533983 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:18:36.534094 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.534021 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:18:36.534094 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.534038 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:18:36.534094 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.534069 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.542227 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.542212 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.542292 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.542241 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-81.us-east-2.compute.internal\": node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:36.557119 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.557099 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:36.627584 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.627533 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal"] Oct 30 23:18:36.627764 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.627641 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:18:36.628644 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.628626 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:18:36.628751 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.628664 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:18:36.628751 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.628679 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:18:36.631018 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.631001 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:18:36.631174 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.631158 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.631219 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.631189 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:18:36.631813 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.631798 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:18:36.631890 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.631837 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:18:36.631890 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.631852 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:18:36.631890 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.631858 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:18:36.631890 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.631878 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:18:36.631890 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.631891 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:18:36.633966 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.633947 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.634048 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.633979 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:18:36.634713 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.634700 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:18:36.634792 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.634730 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:18:36.634792 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.634743 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:18:36.658149 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.658118 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:36.662115 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.662096 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" node="ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.662631 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.662614 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f383e62f25895fcb5217aa3ca64dca1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"f383e62f25895fcb5217aa3ca64dca1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.662679 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.662639 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f383e62f25895fcb5217aa3ca64dca1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"f383e62f25895fcb5217aa3ca64dca1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.662679 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.662658 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70da0cbd0028ec04eeaec823ad5f3973-config\") pod \"kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"70da0cbd0028ec04eeaec823ad5f3973\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.665029 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.665013 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" node="ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.758618 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.758594 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:36.762896 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.762875 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f383e62f25895fcb5217aa3ca64dca1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"f383e62f25895fcb5217aa3ca64dca1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.762973 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.762905 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f383e62f25895fcb5217aa3ca64dca1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"f383e62f25895fcb5217aa3ca64dca1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.762973 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.762922 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70da0cbd0028ec04eeaec823ad5f3973-config\") pod \"kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"70da0cbd0028ec04eeaec823ad5f3973\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.762973 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.762949 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70da0cbd0028ec04eeaec823ad5f3973-config\") pod \"kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"70da0cbd0028ec04eeaec823ad5f3973\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.763079 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.762981 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f383e62f25895fcb5217aa3ca64dca1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"f383e62f25895fcb5217aa3ca64dca1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.763079 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.762980 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f383e62f25895fcb5217aa3ca64dca1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"f383e62f25895fcb5217aa3ca64dca1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.859320 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.859294 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:36.959837 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:36.959797 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:36.965116 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.965095 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:36.968227 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:36.968210 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:37.060787 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:37.060750 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:37.161254 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:37.161179 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:37.261667 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:37.261632 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:37.301942 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.301918 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 23:18:37.302416 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.302101 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 23:18:37.362330 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:37.362298 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:37.444391 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.444308 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:13:37 +0000 UTC" deadline="2028-07-05 18:36:22.190937453 +0000 UTC" Oct 30 23:18:37.444391 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.444344 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14731h17m44.74659732s" Oct 30 23:18:37.462614 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:37.462584 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:37.560059 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.560021 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:18:37.561894 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.561875 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:18:37.562024 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.561906 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:18:37.562024 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.561917 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:18:37.562974 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:37.562956 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:37.659044 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.659014 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:18:37.663042 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:37.663018 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:37.701386 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:37.701307 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70da0cbd0028ec04eeaec823ad5f3973.slice/crio-d6f5c3c59ff333b00077f611922d5d7b2b0d9180a385f4b6d3093dfcf9d53ff2 WatchSource:0}: Error finding container d6f5c3c59ff333b00077f611922d5d7b2b0d9180a385f4b6d3093dfcf9d53ff2: Status 404 returned error can't find the container with id d6f5c3c59ff333b00077f611922d5d7b2b0d9180a385f4b6d3093dfcf9d53ff2 Oct 30 23:18:37.701646 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:37.701618 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf383e62f25895fcb5217aa3ca64dca1a.slice/crio-478334bb3c33b8c957e40eb40e3d3a7bd2b5cd98ddcf604657dc370754314655 WatchSource:0}: Error finding container 478334bb3c33b8c957e40eb40e3d3a7bd2b5cd98ddcf604657dc370754314655: Status 404 returned error can't find the container with id 478334bb3c33b8c957e40eb40e3d3a7bd2b5cd98ddcf604657dc370754314655 Oct 30 23:18:37.706018 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.705991 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:18:37.763523 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:37.763495 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:18:37.828731 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.828702 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:18:37.860934 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.860908 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:37.873803 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.873770 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:18:37.874601 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.874589 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:18:37.882209 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.882195 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:18:37.943100 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:37.943076 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:18:38.336393 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.336330 2581 apiserver.go:52] "Watching apiserver" Oct 30 23:18:38.344071 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.344044 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:18:38.344465 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.344439 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-6sfch","openshift-dns/node-resolver-bwf5h","openshift-multus/multus-nkhth","openshift-network-operator/iptables-alerter-9nwfw","kube-system/konnectivity-agent-89shx","openshift-network-diagnostics/network-check-target-rhm2r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62","openshift-image-registry/node-ca-vqvzf","openshift-multus/network-metrics-daemon-jgq5s","kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-qmn89","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-lwrmd","openshift-ovn-kubernetes/ovnkube-node-562ql"] Oct 30 23:18:38.347285 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.347253 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.349746 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.349723 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.351755 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.351735 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:18:38.351874 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.351838 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kc52k\"" Oct 30 23:18:38.351874 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.351867 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:18:38.352035 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.351924 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:18:38.352684 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.352661 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.353817 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.353797 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qwtvf\"" Oct 30 23:18:38.353922 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.353828 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:18:38.353922 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.353877 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:18:38.353922 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.353904 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:18:38.354093 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.353950 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:18:38.354093 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.353964 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:18:38.356189 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.356161 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:18:38.356774 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.356752 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:18:38.356865 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.356844 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-82xnp\"" Oct 30 23:18:38.357702 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.357683 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-89shx" Oct 30 23:18:38.359827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.359803 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:18:38.359926 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.359863 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 23:18:38.360291 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.360271 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:18:38.360404 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.360386 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:18:38.360587 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.360560 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lwvlw\"" Oct 30 23:18:38.360656 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.360621 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:38.360724 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.360702 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:18:38.361312 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.361294 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qmkhv\"" Oct 30 23:18:38.361473 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.361454 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:18:38.361649 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.361629 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:18:38.363924 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.363903 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:38.364027 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.363981 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:18:38.366455 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.366342 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:38.366455 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.366426 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:18:38.368555 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.368527 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:18:38.371228 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.371211 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.373185 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.372728 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-registration-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.373185 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.372882 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.373185 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.372919 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8mqf\" (UniqueName: \"kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf\") pod \"network-check-target-rhm2r\" (UID: \"a10c4ed2-0361-47e1-8810-8a710b2be42d\") " pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:38.373185 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.373110 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-cnibin\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.373185 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.373159 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-cni-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.374159 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.373562 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-k8s-cni-cncf-io\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.374159 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.373650 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-multus-certs\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.374159 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.373852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9-agent-certs\") pod \"konnectivity-agent-89shx\" (UID: \"d3eeb220-e1ba-46e2-94c8-2bb19e02dde9\") " pod="kube-system/konnectivity-agent-89shx" Oct 30 23:18:38.374159 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.373897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.374159 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.373961 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:38.374159 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.373996 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-system-cni-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.374159 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.374047 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-cni-multus\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.374571 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.374239 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-conf-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.374571 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.374293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97edeeaf-64b2-4628-a669-5fc121c8ee2f-host-slash\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:18:38.374571 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.374349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-system-cni-dir\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.374571 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.374384 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.374571 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.374436 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.374571 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.374517 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-netns\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.374571 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.374552 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-device-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.374903 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.374782 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.374903 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.374818 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-kubelet\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.375000 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.374935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n94jk\" (UniqueName: \"kubernetes.io/projected/29c3d1e4-09b5-4712-ad6d-9a32434def2d-kube-api-access-n94jk\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375107 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97edeeaf-64b2-4628-a669-5fc121c8ee2f-iptables-alerter-script\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375198 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6rsj\" (UniqueName: \"kubernetes.io/projected/97edeeaf-64b2-4628-a669-5fc121c8ee2f-kube-api-access-c6rsj\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375243 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-socket-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375353 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-sys-fs\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7ms7\" (UniqueName: \"kubernetes.io/projected/312d3272-40b0-41c2-8131-bafd6266a11e-kube-api-access-r7ms7\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375416 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-socket-dir-parent\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375488 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-daemon-config\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-etc-kubernetes\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375640 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvc6\" (UniqueName: \"kubernetes.io/projected/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-kube-api-access-kzvc6\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-cnibin\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375732 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-cni-bin\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375787 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-hostroot\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375816 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9-konnectivity-ca\") pod \"konnectivity-agent-89shx\" (UID: \"d3eeb220-e1ba-46e2-94c8-2bb19e02dde9\") " pod="kube-system/konnectivity-agent-89shx" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375853 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375924 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89205e33-cb00-47fe-b78d-cc1a05d1f123-cni-binary-copy\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.376061 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.375986 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhqtj\" (UniqueName: \"kubernetes.io/projected/89205e33-cb00-47fe-b78d-cc1a05d1f123-kube-api-access-xhqtj\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.376955 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.376418 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:18:38.377976 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.377841 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.378668 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.378640 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:18:38.383649 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.383622 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:18:38.402865 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.402834 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mhcmq" Oct 30 23:18:38.416780 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.416682 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mhcmq" Oct 30 23:18:38.460458 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460425 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:18:38.460458 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460449 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:18:38.460807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460477 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:18:38.460807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460425 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:18:38.460807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460481 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:18:38.460807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460425 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:18:38.460807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460452 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k8dwv\"" Oct 30 23:18:38.460807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460520 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:18:38.460807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460542 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:18:38.460807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460548 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:18:38.460807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460559 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:18:38.460807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460722 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:18:38.460807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460754 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:18:38.460807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.460800 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jt8kr\"" Oct 30 23:18:38.461308 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.461283 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:18:38.461431 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.461350 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4bbvk\"" Oct 30 23:18:38.461431 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.461376 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n9xrj\"" Oct 30 23:18:38.461586 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.461533 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:18:38.476809 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.476784 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-cnibin\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.476943 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.476823 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-node-log\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.476943 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.476847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-ovnkube-script-lib\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.476943 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.476867 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-systemd\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.476943 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.476888 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5f12d67-0b46-434f-8c41-a230d421eefd-serviceca\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:18:38.476943 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.476911 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.477224 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.476965 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-cnibin\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.477224 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89205e33-cb00-47fe-b78d-cc1a05d1f123-cni-binary-copy\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.477224 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477064 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.477224 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477158 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-run-netns\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.477224 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477189 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-cnibin\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.477572 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477551 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.477676 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9-agent-certs\") pod \"konnectivity-agent-89shx\" (UID: \"d3eeb220-e1ba-46e2-94c8-2bb19e02dde9\") " pod="kube-system/konnectivity-agent-89shx" Oct 30 23:18:38.477735 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.477735 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477723 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-log-socket\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.477735 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477723 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-cnibin\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.477890 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477867 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxhqr\" (UniqueName: \"kubernetes.io/projected/f5f12d67-0b46-434f-8c41-a230d421eefd-kube-api-access-lxhqr\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:18:38.477945 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477899 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-cni-multus\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.477945 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477927 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.478048 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.477978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-system-cni-dir\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.478048 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478007 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.478048 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478034 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.478202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-device-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.478256 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478224 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-cni-netd\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.478314 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478253 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.478314 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478272 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89205e33-cb00-47fe-b78d-cc1a05d1f123-cni-binary-copy\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.478314 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478282 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97edeeaf-64b2-4628-a669-5fc121c8ee2f-iptables-alerter-script\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:18:38.478314 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478310 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rsj\" (UniqueName: \"kubernetes.io/projected/97edeeaf-64b2-4628-a669-5fc121c8ee2f-kube-api-access-c6rsj\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:18:38.478523 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478337 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysctl-d\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.478523 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478347 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-device-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.478523 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-host\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.478523 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478392 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-socket-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.478523 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478442 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-sys-fs\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.478523 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-kubelet-config\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:38.478846 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478496 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-dbus\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:38.478846 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-socket-dir-parent\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.478846 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478704 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:18:38.478846 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478763 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-etc-kubernetes\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.478846 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478794 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-env-overrides\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.478846 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478820 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19274393-d63a-4889-8c71-0648e21959a5-ovn-node-metrics-cert\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.478846 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478845 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-var-lib-kubelet\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.479208 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvc6\" (UniqueName: \"kubernetes.io/projected/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-kube-api-access-kzvc6\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:38.479208 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478900 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-cni-bin\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.479208 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478947 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-hostroot\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.479208 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.478977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9-konnectivity-ca\") pod \"konnectivity-agent-89shx\" (UID: \"d3eeb220-e1ba-46e2-94c8-2bb19e02dde9\") " pod="kube-system/konnectivity-agent-89shx" Oct 30 23:18:38.479208 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479006 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-etc-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.479208 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479029 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-run\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.479208 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479052 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-sys\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.479208 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479077 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-socket-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.479632 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479266 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhqtj\" (UniqueName: \"kubernetes.io/projected/89205e33-cb00-47fe-b78d-cc1a05d1f123-kube-api-access-xhqtj\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.479632 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-registration-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.479632 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-hosts-file\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:18:38.479632 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479407 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-systemd\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.479632 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479433 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-modprobe-d\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.479632 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479464 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mqf\" (UniqueName: \"kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf\") pod \"network-check-target-rhm2r\" (UID: \"a10c4ed2-0361-47e1-8810-8a710b2be42d\") " pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:38.479632 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-cni-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.479632 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479599 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-k8s-cni-cncf-io\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.479632 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479627 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.480067 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479628 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-multus-certs\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.480067 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479691 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-multus-certs\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.480067 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479738 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-cni-multus\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.480067 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479775 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-system-cni-dir\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.480067 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479833 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.480067 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479902 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.480067 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.479957 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.480387 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480226 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97edeeaf-64b2-4628-a669-5fc121c8ee2f-iptables-alerter-script\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:18:38.480387 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480300 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-var-lib-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.480387 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480334 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.480387 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480363 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-kubernetes\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.480612 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480428 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v9h2\" (UniqueName: \"kubernetes.io/projected/88de885b-31c5-4ce7-a66f-0a212087affd-kube-api-access-7v9h2\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.480612 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480424 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.480612 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480479 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:38.480612 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480506 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-system-cni-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.480612 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480529 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-conf-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.480612 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480585 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-socket-dir-parent\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.480612 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97edeeaf-64b2-4628-a669-5fc121c8ee2f-host-slash\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:18:38.480967 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480622 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2dsq\" (UniqueName: \"kubernetes.io/projected/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-kube-api-access-f2dsq\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:18:38.480967 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480677 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-sys-fs\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.480967 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480678 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-cni-bin\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.480967 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480714 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88de885b-31c5-4ce7-a66f-0a212087affd-tmp\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.480967 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.480839 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:18:38.480967 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.480945 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs podName:8cad31bd-cac3-44ca-bb2c-ca302c4e82e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:38.980915231 +0000 UTC m=+2.934022553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs") pod "network-metrics-daemon-jgq5s" (UID: "8cad31bd-cac3-44ca-bb2c-ca302c4e82e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:18:38.481277 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480998 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-conf-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.481277 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480996 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-cni-bin\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.481277 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.480628 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-etc-kubernetes\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.481277 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481052 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97edeeaf-64b2-4628-a669-5fc121c8ee2f-host-slash\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:18:38.481277 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481069 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-netns\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.481277 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481088 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-registration-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.481277 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481246 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-system-cni-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.481641 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481273 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-slash\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.481641 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481308 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-ovn\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.481641 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481328 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jnn5\" (UniqueName: \"kubernetes.io/projected/19274393-d63a-4889-8c71-0648e21959a5-kube-api-access-9jnn5\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.481641 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481342 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-k8s-cni-cncf-io\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.481641 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481356 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysconfig\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.481641 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481412 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-netns\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.481985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481671 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9-konnectivity-ca\") pod \"konnectivity-agent-89shx\" (UID: \"d3eeb220-e1ba-46e2-94c8-2bb19e02dde9\") " pod="kube-system/konnectivity-agent-89shx" Oct 30 23:18:38.481985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481714 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88de885b-31c5-4ce7-a66f-0a212087affd-etc-tuned\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.481985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481728 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-cni-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.481985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481742 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5f12d67-0b46-434f-8c41-a230d421eefd-host\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:18:38.481985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-kubelet\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.481985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481802 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-hostroot\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.481985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94jk\" (UniqueName: \"kubernetes.io/projected/29c3d1e4-09b5-4712-ad6d-9a32434def2d-kube-api-access-n94jk\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.481985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481902 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-ovnkube-config\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.481985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481922 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-kubelet\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.481985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysctl-conf\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.481985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.481981 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-lib-modules\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.482384 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.482024 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-systemd-units\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.482384 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.482112 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.482384 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.482160 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7ms7\" (UniqueName: \"kubernetes.io/projected/312d3272-40b0-41c2-8131-bafd6266a11e-kube-api-access-r7ms7\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.482384 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.482332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-daemon-config\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.482384 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.482373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-tmp-dir\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:18:38.482614 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.482457 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:38.482614 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.482529 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-kubelet\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.483059 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.483018 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-daemon-config\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.483629 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.483571 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9-agent-certs\") pod \"konnectivity-agent-89shx\" (UID: \"d3eeb220-e1ba-46e2-94c8-2bb19e02dde9\") " pod="kube-system/konnectivity-agent-89shx" Oct 30 23:18:38.492597 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.492522 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:18:38.492597 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.492578 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:18:38.492768 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.492603 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8mqf for pod openshift-network-diagnostics/network-check-target-rhm2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:18:38.492880 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.492821 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf podName:a10c4ed2-0361-47e1-8810-8a710b2be42d nodeName:}" failed. No retries permitted until 2026-10-30 23:18:38.992790281 +0000 UTC m=+2.945897607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k8mqf" (UniqueName: "kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf") pod "network-check-target-rhm2r" (UID: "a10c4ed2-0361-47e1-8810-8a710b2be42d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:18:38.494831 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.494704 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6rsj\" (UniqueName: \"kubernetes.io/projected/97edeeaf-64b2-4628-a669-5fc121c8ee2f-kube-api-access-c6rsj\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:18:38.495228 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.495202 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvc6\" (UniqueName: \"kubernetes.io/projected/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-kube-api-access-kzvc6\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:38.496798 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.496743 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7ms7\" (UniqueName: \"kubernetes.io/projected/312d3272-40b0-41c2-8131-bafd6266a11e-kube-api-access-r7ms7\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.496893 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.496776 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhqtj\" (UniqueName: \"kubernetes.io/projected/89205e33-cb00-47fe-b78d-cc1a05d1f123-kube-api-access-xhqtj\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.497799 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.497773 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94jk\" (UniqueName: \"kubernetes.io/projected/29c3d1e4-09b5-4712-ad6d-9a32434def2d-kube-api-access-n94jk\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.534616 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.534508 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" event={"ID":"70da0cbd0028ec04eeaec823ad5f3973","Type":"ContainerStarted","Data":"d6f5c3c59ff333b00077f611922d5d7b2b0d9180a385f4b6d3093dfcf9d53ff2"} Oct 30 23:18:38.536196 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.536165 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" event={"ID":"f383e62f25895fcb5217aa3ca64dca1a","Type":"ContainerStarted","Data":"478334bb3c33b8c957e40eb40e3d3a7bd2b5cd98ddcf604657dc370754314655"} Oct 30 23:18:38.583724 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.583690 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-kubelet-config\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:38.583917 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.583731 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-dbus\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:38.583917 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.583762 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-env-overrides\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.583917 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.583789 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19274393-d63a-4889-8c71-0648e21959a5-ovn-node-metrics-cert\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.583917 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.583816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-var-lib-kubelet\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.583917 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.583846 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-etc-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.583917 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.583844 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-kubelet-config\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:38.583917 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.583872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-run\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.583917 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.583900 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-sys\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.583927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-hosts-file\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.583967 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-systemd\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.583993 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-modprobe-d\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584037 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-var-lib-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-kubernetes\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9h2\" (UniqueName: \"kubernetes.io/projected/88de885b-31c5-4ce7-a66f-0a212087affd-kube-api-access-7v9h2\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584158 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2dsq\" (UniqueName: \"kubernetes.io/projected/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-kube-api-access-f2dsq\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584175 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-cni-bin\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584190 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88de885b-31c5-4ce7-a66f-0a212087affd-tmp\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584224 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-slash\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584240 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-ovn\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584261 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jnn5\" (UniqueName: \"kubernetes.io/projected/19274393-d63a-4889-8c71-0648e21959a5-kube-api-access-9jnn5\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584313 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-var-lib-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584354 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysconfig\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.584389 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88de885b-31c5-4ce7-a66f-0a212087affd-etc-tuned\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5f12d67-0b46-434f-8c41-a230d421eefd-host\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584454 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-ovnkube-config\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysctl-conf\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584523 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-lib-modules\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584549 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-systemd-units\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584559 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584605 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-tmp-dir\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584628 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-kubernetes\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584632 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584633 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-run\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584646 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-hosts-file\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-kubelet\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584675 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-sys\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584697 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-systemd\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584715 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-var-lib-kubelet\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584742 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-etc-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.585161 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584750 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-node-log\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584796 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-modprobe-d\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584796 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-node-log\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584874 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-dbus\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584899 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysctl-conf\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584930 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-lib-modules\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584933 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-ovnkube-script-lib\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584935 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-cni-bin\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.584972 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-systemd\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585031 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-kubelet\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.585129 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.585197 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret podName:6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f nodeName:}" failed. No retries permitted until 2026-10-30 23:18:39.085178226 +0000 UTC m=+3.038285538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret") pod "global-pull-secret-syncer-6sfch" (UID: "6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585211 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5f12d67-0b46-434f-8c41-a230d421eefd-host\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585268 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysconfig\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585282 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-tmp-dir\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585357 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-slash\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585449 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5f12d67-0b46-434f-8c41-a230d421eefd-serviceca\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585513 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.586339 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585524 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-ovn\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585579 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-systemd-units\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585656 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-systemd\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585688 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-run-netns\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585732 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-run-netns\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585776 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-log-socket\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585810 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhqr\" (UniqueName: \"kubernetes.io/projected/f5f12d67-0b46-434f-8c41-a230d421eefd-kube-api-access-lxhqr\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585910 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-cni-netd\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysctl-d\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.585978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-host\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.586056 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-cni-netd\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.586083 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-ovnkube-config\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.586094 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.586131 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-log-socket\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.586463 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysctl-d\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.586506 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-host\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.586562 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5f12d67-0b46-434f-8c41-a230d421eefd-serviceca\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:18:38.587361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.586982 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19274393-d63a-4889-8c71-0648e21959a5-ovn-node-metrics-cert\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.588190 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.587673 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-env-overrides\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.588190 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.587747 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88de885b-31c5-4ce7-a66f-0a212087affd-tmp\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.588190 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.587924 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-ovnkube-script-lib\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.588190 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.588100 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88de885b-31c5-4ce7-a66f-0a212087affd-etc-tuned\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.597978 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.597903 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jnn5\" (UniqueName: \"kubernetes.io/projected/19274393-d63a-4889-8c71-0648e21959a5-kube-api-access-9jnn5\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.601477 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.601434 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9h2\" (UniqueName: \"kubernetes.io/projected/88de885b-31c5-4ce7-a66f-0a212087affd-kube-api-access-7v9h2\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.603790 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.603742 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2dsq\" (UniqueName: \"kubernetes.io/projected/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-kube-api-access-f2dsq\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:18:38.605335 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.605313 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhqr\" (UniqueName: \"kubernetes.io/projected/f5f12d67-0b46-434f-8c41-a230d421eefd-kube-api-access-lxhqr\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:18:38.668048 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.668016 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:18:38.683749 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:38.683707 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c3d1e4_09b5_4712_ad6d_9a32434def2d.slice/crio-691b50e3328cd30fc1ba9c74703894c9120f2fa9e31c063b43e8ee5d3faecdf9 WatchSource:0}: Error finding container 691b50e3328cd30fc1ba9c74703894c9120f2fa9e31c063b43e8ee5d3faecdf9: Status 404 returned error can't find the container with id 691b50e3328cd30fc1ba9c74703894c9120f2fa9e31c063b43e8ee5d3faecdf9 Oct 30 23:18:38.690119 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.690094 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nkhth" Oct 30 23:18:38.697034 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.696865 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:18:38.699033 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.699001 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:18:38.705406 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.705368 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-89shx" Oct 30 23:18:38.708419 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:38.708388 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312d3272_40b0_41c2_8131_bafd6266a11e.slice/crio-6ae127e8f738069b308baef98763318ed6b2cadc9b6e48d415b365e84a5e5214 WatchSource:0}: Error finding container 6ae127e8f738069b308baef98763318ed6b2cadc9b6e48d415b365e84a5e5214: Status 404 returned error can't find the container with id 6ae127e8f738069b308baef98763318ed6b2cadc9b6e48d415b365e84a5e5214 Oct 30 23:18:38.711150 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:38.711123 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97edeeaf_64b2_4628_a669_5fc121c8ee2f.slice/crio-c076674abdd7778f989e75e87b5b1116454b556044f662cd7c5becc8c740bb6a WatchSource:0}: Error finding container c076674abdd7778f989e75e87b5b1116454b556044f662cd7c5becc8c740bb6a: Status 404 returned error can't find the container with id c076674abdd7778f989e75e87b5b1116454b556044f662cd7c5becc8c740bb6a Oct 30 23:18:38.713466 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.713260 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:18:38.722627 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:38.722596 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3eeb220_e1ba_46e2_94c8_2bb19e02dde9.slice/crio-bcb715c428511e8227d370f524582b4d843b5afaa559b8d68edcce398c9c28c9 WatchSource:0}: Error finding container bcb715c428511e8227d370f524582b4d843b5afaa559b8d68edcce398c9c28c9: Status 404 returned error can't find the container with id bcb715c428511e8227d370f524582b4d843b5afaa559b8d68edcce398c9c28c9 Oct 30 23:18:38.726145 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:38.726110 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecda0b2_bd81_4e60_b31c_ef8fd2d2a22f.slice/crio-ac67a445adc79e1810ead750a60afb6798db06a1d1a18140f437afeed1c1d6ec WatchSource:0}: Error finding container ac67a445adc79e1810ead750a60afb6798db06a1d1a18140f437afeed1c1d6ec: Status 404 returned error can't find the container with id ac67a445adc79e1810ead750a60afb6798db06a1d1a18140f437afeed1c1d6ec Oct 30 23:18:38.740666 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.740634 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:18:38.743299 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.743278 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:18:38.745935 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.745911 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:38.751594 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:38.751544 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88de885b_31c5_4ce7_a66f_0a212087affd.slice/crio-0ea09794cb94f5646655520e6b923547342d9e78012bfc3b7427b22fb0354776 WatchSource:0}: Error finding container 0ea09794cb94f5646655520e6b923547342d9e78012bfc3b7427b22fb0354776: Status 404 returned error can't find the container with id 0ea09794cb94f5646655520e6b923547342d9e78012bfc3b7427b22fb0354776 Oct 30 23:18:38.759993 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:18:38.759938 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f12d67_0b46_434f_8c41_a230d421eefd.slice/crio-1fdfbfc70ccd4ad2fccc4f11f7983ca667f7426522c07035faec6a82cea360e5 WatchSource:0}: Error finding container 1fdfbfc70ccd4ad2fccc4f11f7983ca667f7426522c07035faec6a82cea360e5: Status 404 returned error can't find the container with id 1fdfbfc70ccd4ad2fccc4f11f7983ca667f7426522c07035faec6a82cea360e5 Oct 30 23:18:38.989830 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:38.989725 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:38.990008 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.989856 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:18:38.990008 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:38.989933 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs podName:8cad31bd-cac3-44ca-bb2c-ca302c4e82e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:39.989912531 +0000 UTC m=+3.943019853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs") pod "network-metrics-daemon-jgq5s" (UID: "8cad31bd-cac3-44ca-bb2c-ca302c4e82e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:18:39.090562 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.090526 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mqf\" (UniqueName: \"kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf\") pod \"network-check-target-rhm2r\" (UID: \"a10c4ed2-0361-47e1-8810-8a710b2be42d\") " pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:39.090745 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.090611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:39.090830 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:39.090743 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:18:39.090830 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:39.090769 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:18:39.090830 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:39.090778 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:18:39.090830 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:39.090781 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8mqf for pod openshift-network-diagnostics/network-check-target-rhm2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:18:39.091002 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:39.090835 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret podName:6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f nodeName:}" failed. No retries permitted until 2026-10-30 23:18:40.090816162 +0000 UTC m=+4.043923481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret") pod "global-pull-secret-syncer-6sfch" (UID: "6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:18:39.091002 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:39.090851 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf podName:a10c4ed2-0361-47e1-8810-8a710b2be42d nodeName:}" failed. No retries permitted until 2026-10-30 23:18:40.090844579 +0000 UTC m=+4.043951885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8mqf" (UniqueName: "kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf") pod "network-check-target-rhm2r" (UID: "a10c4ed2-0361-47e1-8810-8a710b2be42d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:18:39.419819 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.419688 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:13:39 +0000 UTC" deadline="2028-07-11 07:29:10.525750758 +0000 UTC" Oct 30 23:18:39.419819 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.419755 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14864h10m31.106000577s" Oct 30 23:18:39.528027 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.527989 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:39.528207 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:39.528178 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:18:39.528649 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.528628 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:39.528756 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:39.528730 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:18:39.551253 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.551195 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" event={"ID":"29c3d1e4-09b5-4712-ad6d-9a32434def2d","Type":"ContainerStarted","Data":"691b50e3328cd30fc1ba9c74703894c9120f2fa9e31c063b43e8ee5d3faecdf9"} Oct 30 23:18:39.567936 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.567835 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qmn89" event={"ID":"88de885b-31c5-4ce7-a66f-0a212087affd","Type":"ContainerStarted","Data":"0ea09794cb94f5646655520e6b923547342d9e78012bfc3b7427b22fb0354776"} Oct 30 23:18:39.570854 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.570826 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9nwfw" event={"ID":"97edeeaf-64b2-4628-a669-5fc121c8ee2f","Type":"ContainerStarted","Data":"c076674abdd7778f989e75e87b5b1116454b556044f662cd7c5becc8c740bb6a"} Oct 30 23:18:39.574458 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.574425 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwrmd" event={"ID":"312d3272-40b0-41c2-8131-bafd6266a11e","Type":"ContainerStarted","Data":"6ae127e8f738069b308baef98763318ed6b2cadc9b6e48d415b365e84a5e5214"} Oct 30 23:18:39.580708 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.580672 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nkhth" event={"ID":"89205e33-cb00-47fe-b78d-cc1a05d1f123","Type":"ContainerStarted","Data":"51f5e9b1a6a87ac27b03fe01101c086817bf1750397b5e8c60565da0b75a4365"} Oct 30 23:18:39.582462 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.582395 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" event={"ID":"19274393-d63a-4889-8c71-0648e21959a5","Type":"ContainerStarted","Data":"0f36e87ad7c41be1adc49a8e7e4530e9b669abac8e1393642a2ab040ec1d662c"} Oct 30 23:18:39.585728 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.585635 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vqvzf" event={"ID":"f5f12d67-0b46-434f-8c41-a230d421eefd","Type":"ContainerStarted","Data":"1fdfbfc70ccd4ad2fccc4f11f7983ca667f7426522c07035faec6a82cea360e5"} Oct 30 23:18:39.588287 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.588207 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bwf5h" event={"ID":"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f","Type":"ContainerStarted","Data":"ac67a445adc79e1810ead750a60afb6798db06a1d1a18140f437afeed1c1d6ec"} Oct 30 23:18:39.591642 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:39.591580 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-89shx" event={"ID":"d3eeb220-e1ba-46e2-94c8-2bb19e02dde9","Type":"ContainerStarted","Data":"bcb715c428511e8227d370f524582b4d843b5afaa559b8d68edcce398c9c28c9"} Oct 30 23:18:40.002346 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:40.001591 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:40.002346 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:40.001927 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:18:40.002346 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:40.001984 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs podName:8cad31bd-cac3-44ca-bb2c-ca302c4e82e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:42.00196931 +0000 UTC m=+5.955076617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs") pod "network-metrics-daemon-jgq5s" (UID: "8cad31bd-cac3-44ca-bb2c-ca302c4e82e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:18:40.102624 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:40.102576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mqf\" (UniqueName: \"kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf\") pod \"network-check-target-rhm2r\" (UID: \"a10c4ed2-0361-47e1-8810-8a710b2be42d\") " pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:40.102832 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:40.102675 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:40.102832 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:40.102824 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:18:40.102959 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:40.102894 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret podName:6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f nodeName:}" failed. No retries permitted until 2026-10-30 23:18:42.102876069 +0000 UTC m=+6.055983381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret") pod "global-pull-secret-syncer-6sfch" (UID: "6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:18:40.103451 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:40.103427 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:18:40.103571 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:40.103455 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:18:40.103571 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:40.103469 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8mqf for pod openshift-network-diagnostics/network-check-target-rhm2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:18:40.103571 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:40.103517 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf podName:a10c4ed2-0361-47e1-8810-8a710b2be42d nodeName:}" failed. No retries permitted until 2026-10-30 23:18:42.103501625 +0000 UTC m=+6.056608934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8mqf" (UniqueName: "kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf") pod "network-check-target-rhm2r" (UID: "a10c4ed2-0361-47e1-8810-8a710b2be42d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:18:40.420096 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:40.420024 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:13:39 +0000 UTC" deadline="2028-05-22 00:47:09.818363239 +0000 UTC" Oct 30 23:18:40.420096 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:40.420069 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13657h28m29.398300178s" Oct 30 23:18:40.531593 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:40.530973 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:40.531593 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:40.531142 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:18:40.624643 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:40.624605 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" event={"ID":"70da0cbd0028ec04eeaec823ad5f3973","Type":"ContainerStarted","Data":"3d2f7238f76a68abcfd417c3c6034a5cc14904cb8e7cc58d20be8766adaad907"} Oct 30 23:18:40.642880 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:40.642817 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" podStartSLOduration=2.6427960539999997 podStartE2EDuration="2.642796054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:40.642213768 +0000 UTC m=+4.595321099" watchObservedRunningTime="2026-10-30 23:18:40.642796054 +0000 UTC m=+4.595903386" Oct 30 23:18:41.527588 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:41.527545 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:41.528096 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:41.527720 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:18:41.528225 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:41.528205 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:41.528390 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:41.528358 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:18:41.641041 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:41.641000 2581 generic.go:356] "Generic (PLEG): container finished" podID="f383e62f25895fcb5217aa3ca64dca1a" containerID="f5c5a3e842ad06feaa5c0c4a9690dd1e8381313d00416c87dceb87359612f142" exitCode=0 Oct 30 23:18:41.642367 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:41.642318 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" event={"ID":"f383e62f25895fcb5217aa3ca64dca1a","Type":"ContainerDied","Data":"f5c5a3e842ad06feaa5c0c4a9690dd1e8381313d00416c87dceb87359612f142"} Oct 30 23:18:42.026568 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:42.025855 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:42.026568 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:42.026053 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:18:42.026568 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:42.026125 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs podName:8cad31bd-cac3-44ca-bb2c-ca302c4e82e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:46.026104433 +0000 UTC m=+9.979211741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs") pod "network-metrics-daemon-jgq5s" (UID: "8cad31bd-cac3-44ca-bb2c-ca302c4e82e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:18:42.126984 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:42.126948 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:42.127119 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:42.127030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mqf\" (UniqueName: \"kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf\") pod \"network-check-target-rhm2r\" (UID: \"a10c4ed2-0361-47e1-8810-8a710b2be42d\") " pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:42.127207 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:42.127190 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:18:42.127270 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:42.127214 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:18:42.127270 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:42.127229 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8mqf for pod openshift-network-diagnostics/network-check-target-rhm2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:18:42.127396 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:42.127285 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf podName:a10c4ed2-0361-47e1-8810-8a710b2be42d nodeName:}" failed. No retries permitted until 2026-10-30 23:18:46.127270437 +0000 UTC m=+10.080377757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8mqf" (UniqueName: "kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf") pod "network-check-target-rhm2r" (UID: "a10c4ed2-0361-47e1-8810-8a710b2be42d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:18:42.127624 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:42.127605 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:18:42.127699 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:42.127665 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret podName:6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f nodeName:}" failed. No retries permitted until 2026-10-30 23:18:46.127650123 +0000 UTC m=+10.080757445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret") pod "global-pull-secret-syncer-6sfch" (UID: "6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:18:42.530382 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:42.530346 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:42.530948 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:42.530494 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:18:42.661568 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:42.661506 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" event={"ID":"f383e62f25895fcb5217aa3ca64dca1a","Type":"ContainerStarted","Data":"491c22ef68df541bd86e95450c1b6daf6619ff2db7c54f9cee88d18817b81e24"} Oct 30 23:18:42.665923 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:42.665886 2581 generic.go:356] "Generic (PLEG): container finished" podID="312d3272-40b0-41c2-8131-bafd6266a11e" containerID="4e3849082a033fe77cfebf63d096e57daec64060097a1f93b0617c4abd84851b" exitCode=0 Oct 30 23:18:42.666066 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:42.665938 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwrmd" event={"ID":"312d3272-40b0-41c2-8131-bafd6266a11e","Type":"ContainerDied","Data":"4e3849082a033fe77cfebf63d096e57daec64060097a1f93b0617c4abd84851b"} Oct 30 23:18:42.689599 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:42.689186 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" podStartSLOduration=4.689168227 podStartE2EDuration="4.689168227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:42.68803612 +0000 UTC m=+6.641143468" watchObservedRunningTime="2026-10-30 23:18:42.689168227 +0000 UTC m=+6.642275558" Oct 30 23:18:43.528340 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:43.527598 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:43.528340 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:43.527747 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:18:43.528340 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:43.528153 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:43.528340 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:43.528243 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:18:43.682327 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:43.682229 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-89shx" event={"ID":"d3eeb220-e1ba-46e2-94c8-2bb19e02dde9","Type":"ContainerStarted","Data":"043dfc4a98e7c3d25342fcba07e34324ad72933294010501eee0db3db1c6cd93"} Oct 30 23:18:43.701705 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:43.701641 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-89shx" podStartSLOduration=2.311106963 podStartE2EDuration="6.701528677s" totalImagesPullingTime="4.390421714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:43.700913988 +0000 UTC m=+7.654021318" watchObservedRunningTime="2026-10-30 23:18:43.701528677 +0000 UTC m=+7.654636011" Oct 30 23:18:44.528010 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:44.527967 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:44.528210 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:44.528127 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:18:44.688030 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:44.687994 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vqvzf" event={"ID":"f5f12d67-0b46-434f-8c41-a230d421eefd","Type":"ContainerStarted","Data":"95ef7f6c7475cc7f585c766d34cfcbfc43ca1ecefa5465393e05e3df177c1927"} Oct 30 23:18:44.709262 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:44.709153 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vqvzf" podStartSLOduration=2.512021315 podStartE2EDuration="7.709133573s" totalImagesPullingTime="5.197112258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:44.708968518 +0000 UTC m=+8.662075850" watchObservedRunningTime="2026-10-30 23:18:44.709133573 +0000 UTC m=+8.662240905" Oct 30 23:18:45.528117 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:45.528029 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:45.528289 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:45.528191 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:18:45.528682 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:45.528657 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:45.528786 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:45.528763 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:18:45.694911 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:45.694832 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" event={"ID":"29c3d1e4-09b5-4712-ad6d-9a32434def2d","Type":"ContainerStarted","Data":"ef3d34d838ce65d2c862d47a6204e8f9216ca3b7dbfb35b6716b7330c29a9165"} Oct 30 23:18:46.070116 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:46.069420 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:46.070116 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:46.069619 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:18:46.070116 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:46.069696 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs podName:8cad31bd-cac3-44ca-bb2c-ca302c4e82e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:54.069675406 +0000 UTC m=+18.022782712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs") pod "network-metrics-daemon-jgq5s" (UID: "8cad31bd-cac3-44ca-bb2c-ca302c4e82e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:18:46.171175 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:46.170165 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:46.171175 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:46.170271 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mqf\" (UniqueName: \"kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf\") pod \"network-check-target-rhm2r\" (UID: \"a10c4ed2-0361-47e1-8810-8a710b2be42d\") " pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:46.171175 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:46.170461 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:18:46.171175 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:46.170486 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:18:46.171175 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:46.170500 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8mqf for pod openshift-network-diagnostics/network-check-target-rhm2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:18:46.171175 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:46.170608 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf podName:a10c4ed2-0361-47e1-8810-8a710b2be42d nodeName:}" failed. No retries permitted until 2026-10-30 23:18:54.170586118 +0000 UTC m=+18.123693442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8mqf" (UniqueName: "kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf") pod "network-check-target-rhm2r" (UID: "a10c4ed2-0361-47e1-8810-8a710b2be42d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:18:46.171175 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:46.171067 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:18:46.171175 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:46.171128 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret podName:6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f nodeName:}" failed. No retries permitted until 2026-10-30 23:18:54.171112342 +0000 UTC m=+18.124219683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret") pod "global-pull-secret-syncer-6sfch" (UID: "6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:18:46.529268 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:46.529230 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:46.529458 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:46.529360 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:18:46.806549 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:46.805486 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-89shx" Oct 30 23:18:46.806549 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:46.806317 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-89shx" Oct 30 23:18:47.433674 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.433624 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:18:47.439444 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.439328 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:18:47.433642271Z","UUID":"47dc4c72-b61f-430b-8811-5856031c2c75","Handler":null,"Name":"","Endpoint":""} Oct 30 23:18:47.442156 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.441732 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:18:47.442156 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.441763 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:18:47.528255 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.528216 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:47.528255 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.528248 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:47.528499 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:47.528373 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:18:47.528818 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:47.528793 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:18:47.701780 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.701741 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" event={"ID":"29c3d1e4-09b5-4712-ad6d-9a32434def2d","Type":"ContainerStarted","Data":"2a5d66974e4610f8cfba0c638bf3e8679c1cb5b9bb4afc436be4d1c5416ecbdb"} Oct 30 23:18:47.703799 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.703701 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nkhth" event={"ID":"89205e33-cb00-47fe-b78d-cc1a05d1f123","Type":"ContainerStarted","Data":"efd912f3deac9319ea5825d99c1f80150b060ad5df48987f4a9a452bef9eb9a9"} Oct 30 23:18:47.706940 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.706915 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bwf5h" event={"ID":"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f","Type":"ContainerStarted","Data":"345c13a03fe5db664bdbd1e04b6ab57b69ea2e8df76c6c2b38c4f6f40634c433"} Oct 30 23:18:47.707255 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.706961 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-89shx" Oct 30 23:18:47.707484 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.707461 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-89shx" Oct 30 23:18:47.728479 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.728431 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nkhth" podStartSLOduration=2.6042986470000002 podStartE2EDuration="10.728418543s" totalImagesPullingTime="8.124119896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:47.727619966 +0000 UTC m=+11.680727297" watchObservedRunningTime="2026-10-30 23:18:47.728418543 +0000 UTC m=+11.681525872" Oct 30 23:18:47.746168 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:47.746120 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bwf5h" podStartSLOduration=2.012083809 podStartE2EDuration="10.746107263s" totalImagesPullingTime="8.734023454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:47.745979425 +0000 UTC m=+11.699086757" watchObservedRunningTime="2026-10-30 23:18:47.746107263 +0000 UTC m=+11.699214592" Oct 30 23:18:48.527588 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:48.527548 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:48.528089 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:48.527709 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:18:48.710534 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:48.710461 2581 generic.go:356] "Generic (PLEG): container finished" podID="312d3272-40b0-41c2-8131-bafd6266a11e" containerID="b7400a62b0383bf3ac4055fb474864d8b2464bb26cc3762fa47eaa493a3fc9be" exitCode=0 Oct 30 23:18:48.710722 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:48.710612 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwrmd" event={"ID":"312d3272-40b0-41c2-8131-bafd6266a11e","Type":"ContainerDied","Data":"b7400a62b0383bf3ac4055fb474864d8b2464bb26cc3762fa47eaa493a3fc9be"} Oct 30 23:18:49.528593 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:49.528372 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:49.528593 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:49.528418 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:49.528593 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:49.528524 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:18:49.529129 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:49.528662 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:18:49.716132 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:49.716092 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" event={"ID":"29c3d1e4-09b5-4712-ad6d-9a32434def2d","Type":"ContainerStarted","Data":"2b44deb055885b061fe7235ae4dbf53e70d57ba09e5684d6edf13363d747f4b1"} Oct 30 23:18:49.717831 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:49.717796 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9nwfw" event={"ID":"97edeeaf-64b2-4628-a669-5fc121c8ee2f","Type":"ContainerStarted","Data":"d49b12d5a47b7e61a9884ae9cd770425320b235ca2b24e9a2e2b5f87d0d76396"} Oct 30 23:18:49.741976 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:49.741926 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podStartSLOduration=2.636681781 podStartE2EDuration="12.741913657s" totalImagesPullingTime="10.105231876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:49.741247136 +0000 UTC m=+13.694354465" watchObservedRunningTime="2026-10-30 23:18:49.741913657 +0000 UTC m=+13.695021003" Oct 30 23:18:49.763372 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:49.763323 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9nwfw" podStartSLOduration=4.109749266 podStartE2EDuration="12.76330966s" totalImagesPullingTime="8.653560394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:49.762735971 +0000 UTC m=+13.715843304" watchObservedRunningTime="2026-10-30 23:18:49.76330966 +0000 UTC m=+13.716416990" Oct 30 23:18:50.528423 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:50.528390 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:50.528602 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:50.528563 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:18:50.722029 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:50.721976 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qmn89" event={"ID":"88de885b-31c5-4ce7-a66f-0a212087affd","Type":"ContainerStarted","Data":"208a3b549775dea0fc552f3a352c74a6772c098e7fba8c64ad8d639ad60e824b"} Oct 30 23:18:50.724777 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:50.724739 2581 generic.go:356] "Generic (PLEG): container finished" podID="312d3272-40b0-41c2-8131-bafd6266a11e" containerID="99e566092093372f977186fb4fd1f21a83de33012e371db21cc72c66b243f209" exitCode=0 Oct 30 23:18:50.724913 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:50.724801 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwrmd" event={"ID":"312d3272-40b0-41c2-8131-bafd6266a11e","Type":"ContainerDied","Data":"99e566092093372f977186fb4fd1f21a83de33012e371db21cc72c66b243f209"} Oct 30 23:18:50.758830 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:50.758780 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qmn89" podStartSLOduration=2.499402988 podStartE2EDuration="13.758763694s" totalImagesPullingTime="11.259360706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:50.758677669 +0000 UTC m=+14.711785049" watchObservedRunningTime="2026-10-30 23:18:50.758763694 +0000 UTC m=+14.711871022" Oct 30 23:18:51.528408 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:51.528373 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:51.528604 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:51.528518 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:18:51.528604 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:51.528575 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:51.529085 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:51.528698 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:18:52.528125 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:52.527895 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:52.528291 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:52.528240 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:18:52.732292 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:52.732254 2581 generic.go:356] "Generic (PLEG): container finished" podID="312d3272-40b0-41c2-8131-bafd6266a11e" containerID="2e1202958373792a5345354eba674fe996b22e7e2dc1757e458a63b4655ef97e" exitCode=0 Oct 30 23:18:52.732747 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:52.732324 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwrmd" event={"ID":"312d3272-40b0-41c2-8131-bafd6266a11e","Type":"ContainerDied","Data":"2e1202958373792a5345354eba674fe996b22e7e2dc1757e458a63b4655ef97e"} Oct 30 23:18:53.528149 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:53.528111 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:53.528323 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:53.528248 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:18:53.528323 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:53.528307 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:53.528410 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:53.528366 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:18:54.133391 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:54.133349 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:54.133797 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:54.133539 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:18:54.133797 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:54.133609 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs podName:8cad31bd-cac3-44ca-bb2c-ca302c4e82e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:10.133594559 +0000 UTC m=+34.086701863 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs") pod "network-metrics-daemon-jgq5s" (UID: "8cad31bd-cac3-44ca-bb2c-ca302c4e82e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:18:54.234165 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:54.234121 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mqf\" (UniqueName: \"kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf\") pod \"network-check-target-rhm2r\" (UID: \"a10c4ed2-0361-47e1-8810-8a710b2be42d\") " pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:54.234353 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:54.234195 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:54.234353 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:54.234299 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:18:54.234353 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:54.234312 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:18:54.234353 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:54.234322 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:18:54.234353 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:54.234333 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8mqf for pod openshift-network-diagnostics/network-check-target-rhm2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:18:54.234587 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:54.234384 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf podName:a10c4ed2-0361-47e1-8810-8a710b2be42d nodeName:}" failed. No retries permitted until 2026-10-30 23:19:10.234363291 +0000 UTC m=+34.187470618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8mqf" (UniqueName: "kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf") pod "network-check-target-rhm2r" (UID: "a10c4ed2-0361-47e1-8810-8a710b2be42d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:18:54.234587 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:54.234409 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret podName:6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f nodeName:}" failed. No retries permitted until 2026-10-30 23:19:10.234391568 +0000 UTC m=+34.187498893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret") pod "global-pull-secret-syncer-6sfch" (UID: "6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:18:54.529834 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:54.529584 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:54.529976 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:54.529951 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:18:54.740509 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:54.740453 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" event={"ID":"19274393-d63a-4889-8c71-0648e21959a5","Type":"ContainerStarted","Data":"3a0b7f0fe5f30176b5c2b926930f0e9c3773e1f01096208c6f45acbfc230c6d9"} Oct 30 23:18:54.740640 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:54.740564 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" event={"ID":"19274393-d63a-4889-8c71-0648e21959a5","Type":"ContainerStarted","Data":"68f0fd7fe90beda6b7bd582e59f32b88031f741386e336c3d2a5fc0b4b028edd"} Oct 30 23:18:54.740640 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:54.740597 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" event={"ID":"19274393-d63a-4889-8c71-0648e21959a5","Type":"ContainerStarted","Data":"d05c8410a779e7fec8f592874c00585f83430153fb5d8dc402da7189722a85a0"} Oct 30 23:18:54.740640 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:54.740626 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" event={"ID":"19274393-d63a-4889-8c71-0648e21959a5","Type":"ContainerStarted","Data":"26eed51551ff9a98b96d012f6230ef9c66a16fecc15557c13ff1e3c0fddb40e9"} Oct 30 23:18:54.740785 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:54.740660 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" event={"ID":"19274393-d63a-4889-8c71-0648e21959a5","Type":"ContainerStarted","Data":"a47f76e19cbf0cd34adcf34bbfa96b1a04016b97d3dcbba045fdbd3d78c3c04e"} Oct 30 23:18:54.740785 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:54.740688 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" event={"ID":"19274393-d63a-4889-8c71-0648e21959a5","Type":"ContainerStarted","Data":"05957fa785d3bb139664fcc57ca72475fbc26fe9bed533cd13fcb87b26773ac3"} Oct 30 23:18:54.743112 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:54.743084 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwrmd" event={"ID":"312d3272-40b0-41c2-8131-bafd6266a11e","Type":"ContainerStarted","Data":"badc30416bac8cd0c51aeed0d778396004f52a7b54a351918962dca91433b3f7"} Oct 30 23:18:55.528314 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:55.528279 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:55.528314 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:55.528312 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:55.528748 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:55.528388 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:18:55.528748 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:55.528527 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:18:55.749637 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:55.749600 2581 generic.go:356] "Generic (PLEG): container finished" podID="312d3272-40b0-41c2-8131-bafd6266a11e" containerID="badc30416bac8cd0c51aeed0d778396004f52a7b54a351918962dca91433b3f7" exitCode=0 Oct 30 23:18:55.749780 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:55.749658 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwrmd" event={"ID":"312d3272-40b0-41c2-8131-bafd6266a11e","Type":"ContainerDied","Data":"badc30416bac8cd0c51aeed0d778396004f52a7b54a351918962dca91433b3f7"} Oct 30 23:18:56.529043 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:56.529009 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:56.529622 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:56.529129 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:18:56.755805 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:56.755772 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" event={"ID":"19274393-d63a-4889-8c71-0648e21959a5","Type":"ContainerStarted","Data":"57c9e11e3671cbbfd2c731163023febe2b6e3f4d42d12fb2b3c1029eef76c721"} Oct 30 23:18:56.759846 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:56.759814 2581 generic.go:356] "Generic (PLEG): container finished" podID="312d3272-40b0-41c2-8131-bafd6266a11e" containerID="16c4d84931e1f719b58d9f373c0f409af8bf57fca3c72b5cc7c217668206c4b1" exitCode=0 Oct 30 23:18:56.759932 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:56.759854 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwrmd" event={"ID":"312d3272-40b0-41c2-8131-bafd6266a11e","Type":"ContainerDied","Data":"16c4d84931e1f719b58d9f373c0f409af8bf57fca3c72b5cc7c217668206c4b1"} Oct 30 23:18:57.528064 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:57.528029 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:57.528064 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:57.528069 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:57.528303 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:57.528135 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:18:57.528303 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:57.528246 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:18:57.765240 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:57.765205 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwrmd" event={"ID":"312d3272-40b0-41c2-8131-bafd6266a11e","Type":"ContainerStarted","Data":"5708b86d1a113e4bac887d8bef348bfec1cffb2a51be499cb410926055f1c990"} Oct 30 23:18:57.797180 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:57.794853 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lwrmd" podStartSLOduration=8.259650205 podStartE2EDuration="20.794811568s" totalImagesPullingTime="12.535161363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:57.792745666 +0000 UTC m=+21.745853005" watchObservedRunningTime="2026-10-30 23:18:57.794811568 +0000 UTC m=+21.747918892" Oct 30 23:18:58.527568 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:58.527532 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:18:58.527806 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:58.527682 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:18:59.527730 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:59.527693 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:18:59.527730 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:59.527735 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:18:59.528539 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:59.527816 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:18:59.528539 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:18:59.527955 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:18:59.772682 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:59.772484 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" event={"ID":"19274393-d63a-4889-8c71-0648e21959a5","Type":"ContainerStarted","Data":"1d36cf57fc43656bb45a7cf70d5ebff3eaf868cded08fc853463cc66b7cc2b3e"} Oct 30 23:18:59.772846 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:59.772731 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:59.789037 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:59.788955 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:18:59.806809 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:18:59.806772 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" podStartSLOduration=7.279328276 podStartE2EDuration="22.80676063s" totalImagesPullingTime="15.527432354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:59.805254414 +0000 UTC m=+23.758361740" watchObservedRunningTime="2026-10-30 23:18:59.80676063 +0000 UTC m=+23.759867977" Oct 30 23:19:00.528316 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:00.528279 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:00.528736 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:00.528428 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:00.774765 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:00.774741 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:19:00.775166 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:00.775151 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:00.791150 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:00.791084 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:01.528371 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:01.528339 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:01.528802 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:01.528447 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:01.528802 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:01.528500 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:01.528802 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:01.528583 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:01.776531 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:01.776505 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:19:02.527980 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:02.527949 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:02.528162 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:02.528085 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:02.778319 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:02.778246 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:19:03.150827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:03.150797 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:03.160965 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:03.160928 2581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" podUID="19274393-d63a-4889-8c71-0648e21959a5" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 23:19:03.169663 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:03.169636 2581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" podUID="19274393-d63a-4889-8c71-0648e21959a5" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 23:19:03.527599 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:03.527526 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:03.527767 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:03.527525 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:03.527767 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:03.527613 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:03.527767 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:03.527693 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:03.789918 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:03.789835 2581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" podUID="19274393-d63a-4889-8c71-0648e21959a5" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 23:19:04.527564 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:04.527530 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:04.527728 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:04.527644 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:05.103926 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:05.103897 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bwf5h_1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f/dns-node-resolver/0.log" Oct 30 23:19:05.528418 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:05.528339 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:05.528418 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:05.528366 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:05.528623 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:05.528453 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:05.528623 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:05.528569 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:06.279371 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:06.279347 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vqvzf_f5f12d67-0b46-434f-8c41-a230d421eefd/node-ca/0.log" Oct 30 23:19:06.529636 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:06.529551 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:06.529778 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:06.529680 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:07.528085 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:07.528048 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:07.528610 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:07.528048 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:07.528610 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:07.528187 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:07.528610 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:07.528214 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:08.528404 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:08.528372 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:08.528851 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:08.528523 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:09.528094 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:09.528050 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:09.528094 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:09.528095 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:09.528306 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:09.528169 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:09.528306 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:09.528288 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:10.146164 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:10.146137 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:10.146607 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:10.146253 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:19:10.146607 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:10.146301 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs podName:8cad31bd-cac3-44ca-bb2c-ca302c4e82e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:42.146287894 +0000 UTC m=+66.099395198 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs") pod "network-metrics-daemon-jgq5s" (UID: "8cad31bd-cac3-44ca-bb2c-ca302c4e82e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:19:10.246807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:10.246785 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mqf\" (UniqueName: \"kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf\") pod \"network-check-target-rhm2r\" (UID: \"a10c4ed2-0361-47e1-8810-8a710b2be42d\") " pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:10.247045 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:10.246831 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:10.247045 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:10.246994 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:19:10.247045 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:10.247014 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:19:10.247045 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:10.247018 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:19:10.247045 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:10.247025 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8mqf for pod openshift-network-diagnostics/network-check-target-rhm2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:19:10.247315 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:10.247060 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret podName:6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f nodeName:}" failed. No retries permitted until 2026-10-30 23:19:42.247047913 +0000 UTC m=+66.200155216 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret") pod "global-pull-secret-syncer-6sfch" (UID: "6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:19:10.247315 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:10.247073 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf podName:a10c4ed2-0361-47e1-8810-8a710b2be42d nodeName:}" failed. No retries permitted until 2026-10-30 23:19:42.247066864 +0000 UTC m=+66.200174167 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8mqf" (UniqueName: "kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf") pod "network-check-target-rhm2r" (UID: "a10c4ed2-0361-47e1-8810-8a710b2be42d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:19:10.528237 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:10.528144 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:10.528401 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:10.528298 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:11.528069 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:11.528032 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:11.528479 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:11.528160 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:11.528479 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:11.528188 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:11.528479 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:11.528232 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:12.528528 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:12.528489 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:12.529058 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:12.528620 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:13.528234 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:13.528201 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:13.528234 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:13.528240 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:13.528438 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:13.528323 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:13.528506 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:13.528437 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:14.527398 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:14.527371 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:14.527856 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:14.527480 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:15.527626 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:15.527598 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:15.527998 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:15.527710 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:15.527998 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:15.527764 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:15.527998 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:15.527842 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:16.530711 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:16.530679 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:16.531080 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:16.530778 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:17.528325 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:17.528272 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:17.528527 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:17.528436 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:17.528527 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:17.528480 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:17.528601 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:17.528565 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:18.527673 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:18.527631 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:18.528167 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:18.527739 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:19.528219 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:19.528174 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:19.528632 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:19.528292 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:19.528632 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:19.528343 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:19.528632 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:19.528425 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:20.528270 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:20.528238 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:20.528677 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:20.528343 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:21.497525 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:21.497487 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jgq5s"] Oct 30 23:19:21.497867 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:21.497842 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:21.497963 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:21.497947 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:21.500202 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:21.500142 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rhm2r"] Oct 30 23:19:21.500553 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:21.500496 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:21.500798 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:21.500767 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:21.502599 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:21.501662 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6sfch"] Oct 30 23:19:21.502599 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:21.501750 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:21.502599 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:21.501835 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:22.438136 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:22.437977 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bwf5h_1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f/dns-node-resolver/0.log" Oct 30 23:19:23.527758 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:23.527719 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:23.527758 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:23.527742 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:23.528196 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:23.527835 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:23.528196 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:23.527849 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:23.528196 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:23.527932 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:23.528196 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:23.527968 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:23.837960 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:23.837937 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vqvzf_f5f12d67-0b46-434f-8c41-a230d421eefd/node-ca/0.log" Oct 30 23:19:25.528227 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:25.528188 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:25.528727 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:25.528269 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:25.528727 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:25.528356 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:25.528727 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:25.528364 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6sfch" podUID="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" Oct 30 23:19:25.528727 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:25.528491 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgq5s" podUID="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" Oct 30 23:19:25.528727 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:25.528561 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" Oct 30 23:19:26.307177 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.307147 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeReady" Oct 30 23:19:26.307356 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.307310 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 23:19:26.356511 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.356449 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h4hmx"] Oct 30 23:19:26.359434 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.359409 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.361663 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.361639 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9tcb2"] Oct 30 23:19:26.363843 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.363824 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zcrv2\"" Oct 30 23:19:26.364067 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.364050 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:19:26.364119 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.364110 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tcb2" Oct 30 23:19:26.365628 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.365614 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:19:26.374838 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.374817 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h4hmx"] Oct 30 23:19:26.376712 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.376690 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tcb2"] Oct 30 23:19:26.456927 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.456895 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sjkl\" (UniqueName: \"kubernetes.io/projected/cb76bb5e-c0a6-4a38-8732-40193dd39cac-kube-api-access-9sjkl\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.456927 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.456931 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb76bb5e-c0a6-4a38-8732-40193dd39cac-metrics-tls\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.457115 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.457064 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd7vj\" (UniqueName: \"kubernetes.io/projected/284dba08-fe12-4942-a5fd-6c1e2f979a17-kube-api-access-kd7vj\") pod \"ingress-canary-9tcb2\" (UID: \"284dba08-fe12-4942-a5fd-6c1e2f979a17\") " pod="openshift-ingress-canary/ingress-canary-9tcb2" Oct 30 23:19:26.457149 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.457112 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/284dba08-fe12-4942-a5fd-6c1e2f979a17-cert\") pod \"ingress-canary-9tcb2\" (UID: \"284dba08-fe12-4942-a5fd-6c1e2f979a17\") " pod="openshift-ingress-canary/ingress-canary-9tcb2" Oct 30 23:19:26.457149 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.457129 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb76bb5e-c0a6-4a38-8732-40193dd39cac-tmp-dir\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.457208 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.457157 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb76bb5e-c0a6-4a38-8732-40193dd39cac-config-volume\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.463396 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.463373 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7vq6t"] Oct 30 23:19:26.464001 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.463983 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:19:26.464282 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.464267 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gskv9\"" Oct 30 23:19:26.466206 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.466193 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.466674 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.466656 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:19:26.466788 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.466737 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:19:26.469943 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.469928 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:19:26.470027 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.470008 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:19:26.470067 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.470046 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:19:26.479538 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.479492 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7vq6t"] Oct 30 23:19:26.557768 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.557734 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd7vj\" (UniqueName: \"kubernetes.io/projected/284dba08-fe12-4942-a5fd-6c1e2f979a17-kube-api-access-kd7vj\") pod \"ingress-canary-9tcb2\" (UID: \"284dba08-fe12-4942-a5fd-6c1e2f979a17\") " pod="openshift-ingress-canary/ingress-canary-9tcb2" Oct 30 23:19:26.557768 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.557772 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/284dba08-fe12-4942-a5fd-6c1e2f979a17-cert\") pod \"ingress-canary-9tcb2\" (UID: \"284dba08-fe12-4942-a5fd-6c1e2f979a17\") " pod="openshift-ingress-canary/ingress-canary-9tcb2" Oct 30 23:19:26.558345 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.557798 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb76bb5e-c0a6-4a38-8732-40193dd39cac-tmp-dir\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.558345 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.557826 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d54e0641-5f0d-41bd-8b1b-900463a24cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.558345 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.557870 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb76bb5e-c0a6-4a38-8732-40193dd39cac-config-volume\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.558345 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.557897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d54e0641-5f0d-41bd-8b1b-900463a24cfa-crio-socket\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.558345 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.557923 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sjkl\" (UniqueName: \"kubernetes.io/projected/cb76bb5e-c0a6-4a38-8732-40193dd39cac-kube-api-access-9sjkl\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.558345 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.558006 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d54e0641-5f0d-41bd-8b1b-900463a24cfa-data-volume\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.558345 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.558061 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb76bb5e-c0a6-4a38-8732-40193dd39cac-metrics-tls\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.558345 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.558146 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d54e0641-5f0d-41bd-8b1b-900463a24cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.558345 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.558183 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76js\" (UniqueName: \"kubernetes.io/projected/d54e0641-5f0d-41bd-8b1b-900463a24cfa-kube-api-access-c76js\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.558345 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.558244 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb76bb5e-c0a6-4a38-8732-40193dd39cac-tmp-dir\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.558673 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.558588 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb76bb5e-c0a6-4a38-8732-40193dd39cac-config-volume\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.561882 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.561858 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb76bb5e-c0a6-4a38-8732-40193dd39cac-metrics-tls\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.561972 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.561899 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/284dba08-fe12-4942-a5fd-6c1e2f979a17-cert\") pod \"ingress-canary-9tcb2\" (UID: \"284dba08-fe12-4942-a5fd-6c1e2f979a17\") " pod="openshift-ingress-canary/ingress-canary-9tcb2" Oct 30 23:19:26.564713 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.564696 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z9rpn\"" Oct 30 23:19:26.564762 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.564696 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:19:26.569851 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.569824 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd7vj\" (UniqueName: \"kubernetes.io/projected/284dba08-fe12-4942-a5fd-6c1e2f979a17-kube-api-access-kd7vj\") pod \"ingress-canary-9tcb2\" (UID: \"284dba08-fe12-4942-a5fd-6c1e2f979a17\") " pod="openshift-ingress-canary/ingress-canary-9tcb2" Oct 30 23:19:26.570046 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.570027 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sjkl\" (UniqueName: \"kubernetes.io/projected/cb76bb5e-c0a6-4a38-8732-40193dd39cac-kube-api-access-9sjkl\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.659361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.659325 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d54e0641-5f0d-41bd-8b1b-900463a24cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.659361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.659369 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d54e0641-5f0d-41bd-8b1b-900463a24cfa-crio-socket\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.659606 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.659389 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d54e0641-5f0d-41bd-8b1b-900463a24cfa-data-volume\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.659606 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.659426 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d54e0641-5f0d-41bd-8b1b-900463a24cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.659606 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.659442 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76js\" (UniqueName: \"kubernetes.io/projected/d54e0641-5f0d-41bd-8b1b-900463a24cfa-kube-api-access-c76js\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.659606 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.659505 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d54e0641-5f0d-41bd-8b1b-900463a24cfa-crio-socket\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.659882 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.659859 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d54e0641-5f0d-41bd-8b1b-900463a24cfa-data-volume\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.660129 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.660074 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d54e0641-5f0d-41bd-8b1b-900463a24cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.661819 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.661796 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d54e0641-5f0d-41bd-8b1b-900463a24cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.674451 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.674428 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:26.677918 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.677903 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tcb2" Oct 30 23:19:26.680271 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.680247 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76js\" (UniqueName: \"kubernetes.io/projected/d54e0641-5f0d-41bd-8b1b-900463a24cfa-kube-api-access-c76js\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.781379 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.781348 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:26.967027 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.966998 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tcb2"] Oct 30 23:19:26.969848 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:26.969824 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h4hmx"] Oct 30 23:19:26.970884 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:26.970858 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284dba08_fe12_4942_a5fd_6c1e2f979a17.slice/crio-52d2af34c7e2a0cad4531f75b651b3cca2d620973340c95b09ae8f0c140747d7 WatchSource:0}: Error finding container 52d2af34c7e2a0cad4531f75b651b3cca2d620973340c95b09ae8f0c140747d7: Status 404 returned error can't find the container with id 52d2af34c7e2a0cad4531f75b651b3cca2d620973340c95b09ae8f0c140747d7 Oct 30 23:19:26.973143 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:26.973120 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb76bb5e_c0a6_4a38_8732_40193dd39cac.slice/crio-ac9e25b291a629c4f62445e1df4eb80cdb346647f0f093e8a5ad095dfee7e1d7 WatchSource:0}: Error finding container ac9e25b291a629c4f62445e1df4eb80cdb346647f0f093e8a5ad095dfee7e1d7: Status 404 returned error can't find the container with id ac9e25b291a629c4f62445e1df4eb80cdb346647f0f093e8a5ad095dfee7e1d7 Oct 30 23:19:27.052847 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.052781 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7vq6t"] Oct 30 23:19:27.055511 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:27.055484 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54e0641_5f0d_41bd_8b1b_900463a24cfa.slice/crio-1a9d86c3ed079b5ced4cf62f22b432c8504dcc3514cfb96275b525e67b595ab6 WatchSource:0}: Error finding container 1a9d86c3ed079b5ced4cf62f22b432c8504dcc3514cfb96275b525e67b595ab6: Status 404 returned error can't find the container with id 1a9d86c3ed079b5ced4cf62f22b432c8504dcc3514cfb96275b525e67b595ab6 Oct 30 23:19:27.528274 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.528242 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:27.528445 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.528298 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:27.528445 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.528401 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:27.564117 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.564090 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:19:27.574028 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.574001 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:19:27.581775 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.581759 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-djdr2\"" Oct 30 23:19:27.581830 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.581773 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:19:27.581830 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.581762 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9tmfn\"" Oct 30 23:19:27.581830 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.581762 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2nz9z\"" Oct 30 23:19:27.581830 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.581808 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:19:27.830077 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.830039 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tcb2" event={"ID":"284dba08-fe12-4942-a5fd-6c1e2f979a17","Type":"ContainerStarted","Data":"52d2af34c7e2a0cad4531f75b651b3cca2d620973340c95b09ae8f0c140747d7"} Oct 30 23:19:27.832166 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.832090 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vq6t" event={"ID":"d54e0641-5f0d-41bd-8b1b-900463a24cfa","Type":"ContainerStarted","Data":"2b378ea79fe61c9cfa36558baa8822914b55f7bee7c80bb9ada1076906d16f81"} Oct 30 23:19:27.832166 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.832138 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vq6t" event={"ID":"d54e0641-5f0d-41bd-8b1b-900463a24cfa","Type":"ContainerStarted","Data":"1a9d86c3ed079b5ced4cf62f22b432c8504dcc3514cfb96275b525e67b595ab6"} Oct 30 23:19:27.833742 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:27.833712 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h4hmx" event={"ID":"cb76bb5e-c0a6-4a38-8732-40193dd39cac","Type":"ContainerStarted","Data":"ac9e25b291a629c4f62445e1df4eb80cdb346647f0f093e8a5ad095dfee7e1d7"} Oct 30 23:19:28.480560 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.479435 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb"] Oct 30 23:19:28.482935 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.482912 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.498307 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.498283 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:19:28.500768 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.500748 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xj5wh\"" Oct 30 23:19:28.502055 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.502035 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:19:28.502192 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.502091 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:19:28.502192 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.502113 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:19:28.506377 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.506355 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb"] Oct 30 23:19:28.514767 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.514742 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hq9sq"] Oct 30 23:19:28.517799 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.517780 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.520432 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.520409 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4"] Oct 30 23:19:28.523667 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.523650 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.538916 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.538882 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4"] Oct 30 23:19:28.573522 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.573489 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-tls\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.573984 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.573539 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0371e4c-ff4b-4867-98a1-bb598ab03845-metrics-client-ca\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.573984 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.573576 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.573984 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.573604 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm2q8\" (UniqueName: \"kubernetes.io/projected/68aefc1f-0a32-4722-a494-054f46f44b5b-kube-api-access-mm2q8\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.573984 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.573734 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-sys\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.573984 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.573927 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-wtmp\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.574233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574002 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-root\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.574233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574029 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.574233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574078 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtnkq\" (UniqueName: \"kubernetes.io/projected/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-api-access-qtnkq\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.574233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574135 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.574233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68aefc1f-0a32-4722-a494-054f46f44b5b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.574479 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574251 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-accelerators-collector-config\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.574479 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574275 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.574479 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574303 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68aefc1f-0a32-4722-a494-054f46f44b5b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.574479 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574330 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-textfile\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.574479 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574357 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdzpm\" (UniqueName: \"kubernetes.io/projected/a0371e4c-ff4b-4867-98a1-bb598ab03845-kube-api-access-mdzpm\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.574479 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574394 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/68aefc1f-0a32-4722-a494-054f46f44b5b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.574479 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574417 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.574479 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.574447 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.588839 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.588680 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 23:19:28.588839 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.588692 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:19:28.588839 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.588680 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:19:28.588839 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.588722 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 23:19:28.588839 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.588772 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8bg6s\"" Oct 30 23:19:28.588839 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.588813 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hgj94\"" Oct 30 23:19:28.589155 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.588972 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:19:28.601407 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.601376 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 23:19:28.601570 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.601413 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:19:28.675732 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.675690 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-sys\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.675895 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.675759 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-wtmp\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.675895 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.675789 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-root\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.675895 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.675816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.675895 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.675833 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtnkq\" (UniqueName: \"kubernetes.io/projected/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-api-access-qtnkq\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.675895 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.675867 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.676082 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.675897 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68aefc1f-0a32-4722-a494-054f46f44b5b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.676082 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.675939 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-accelerators-collector-config\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.676082 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.675992 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.676082 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676011 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68aefc1f-0a32-4722-a494-054f46f44b5b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.676082 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676037 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-textfile\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.676082 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676064 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdzpm\" (UniqueName: \"kubernetes.io/projected/a0371e4c-ff4b-4867-98a1-bb598ab03845-kube-api-access-mdzpm\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.676266 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676089 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/68aefc1f-0a32-4722-a494-054f46f44b5b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.676266 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676108 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.676266 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676131 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.676266 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676154 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-tls\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.676266 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0371e4c-ff4b-4867-98a1-bb598ab03845-metrics-client-ca\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.676266 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676215 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.676266 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676243 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm2q8\" (UniqueName: \"kubernetes.io/projected/68aefc1f-0a32-4722-a494-054f46f44b5b-kube-api-access-mm2q8\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.676663 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676633 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-sys\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.676812 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676795 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-wtmp\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.676856 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.676845 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-root\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.677623 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.677580 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.678828 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.678804 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-textfile\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.678936 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.678829 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.678999 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.678949 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68aefc1f-0a32-4722-a494-054f46f44b5b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.678999 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.678985 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.679110 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:28.679084 2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Oct 30 23:19:28.679160 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:28.679141 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-tls podName:3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:29.179123679 +0000 UTC m=+53.132230999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-tls") pod "kube-state-metrics-5dc8d7d89b-thls4" (UID: "3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0") : secret "kube-state-metrics-tls" not found Oct 30 23:19:28.679636 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.679582 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0371e4c-ff4b-4867-98a1-bb598ab03845-metrics-client-ca\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.680785 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.680725 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-accelerators-collector-config\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.681515 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.681490 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.681900 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.681850 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/68aefc1f-0a32-4722-a494-054f46f44b5b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.682278 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.682254 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.682360 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.682312 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68aefc1f-0a32-4722-a494-054f46f44b5b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.683099 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.683076 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-tls\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.704300 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.700301 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm2q8\" (UniqueName: \"kubernetes.io/projected/68aefc1f-0a32-4722-a494-054f46f44b5b-kube-api-access-mm2q8\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.708900 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.708873 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtnkq\" (UniqueName: \"kubernetes.io/projected/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-api-access-qtnkq\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:28.709680 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.709582 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdzpm\" (UniqueName: \"kubernetes.io/projected/a0371e4c-ff4b-4867-98a1-bb598ab03845-kube-api-access-mdzpm\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.802772 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.802269 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:28.835953 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.835727 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:28.843142 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.843111 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vq6t" event={"ID":"d54e0641-5f0d-41bd-8b1b-900463a24cfa","Type":"ContainerStarted","Data":"53c937ba1ba3346bb3ae7bace3828d244f135c4939b59c6449fa3db6c81aa0a2"} Oct 30 23:19:28.853259 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:28.853223 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0371e4c_ff4b_4867_98a1_bb598ab03845.slice/crio-ecbb743dd6362c36741ee41c3b8ae2d115ec350e24a5567b3b67d3251a24ffd3 WatchSource:0}: Error finding container ecbb743dd6362c36741ee41c3b8ae2d115ec350e24a5567b3b67d3251a24ffd3: Status 404 returned error can't find the container with id ecbb743dd6362c36741ee41c3b8ae2d115ec350e24a5567b3b67d3251a24ffd3 Oct 30 23:19:28.876220 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.876033 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-586c8ddf94-s29tc"] Oct 30 23:19:28.882982 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.881369 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:28.887006 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.886613 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 23:19:28.889123 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.889098 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 23:19:28.889269 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.889250 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 23:19:28.889422 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.889405 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 23:19:28.890594 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.890574 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 23:19:28.895668 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.895436 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tr4dk\"" Oct 30 23:19:28.899487 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.899409 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586c8ddf94-s29tc"] Oct 30 23:19:28.901050 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.901029 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 23:19:28.980834 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.980800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5349bf3-d267-4d03-8f45-60e58c5f881e-tmp\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:28.981002 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.980851 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:28.981002 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.980879 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:28.981096 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.981041 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hznz\" (UniqueName: \"kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:28.981096 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.981084 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:28.981193 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.981116 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:28.981245 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.981227 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:28.981298 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.981255 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:28.987150 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.987116 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 23:19:28.987301 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:28.987277 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 23:19:29.082955 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.082768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5349bf3-d267-4d03-8f45-60e58c5f881e-tmp\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.082955 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.082838 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.082955 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.082872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.082955 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.082909 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hznz\" (UniqueName: \"kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.082955 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.082941 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.083338 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.082968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.083395 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.083013 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.083449 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.083431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.084332 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.084282 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.084474 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.084396 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.084474 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.084404 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.084474 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.084439 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.084627 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.084605 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5349bf3-d267-4d03-8f45-60e58c5f881e-tmp\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.087278 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.087259 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.092251 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.092228 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.098655 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.096416 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hznz\" (UniqueName: \"kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.098845 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.098822 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb"] Oct 30 23:19:29.103677 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:29.103652 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68aefc1f_0a32_4722_a494_054f46f44b5b.slice/crio-150395e7507b6aff8a2de8ef1d74767338f9a3bd6850740682852f5395c16d64 WatchSource:0}: Error finding container 150395e7507b6aff8a2de8ef1d74767338f9a3bd6850740682852f5395c16d64: Status 404 returned error can't find the container with id 150395e7507b6aff8a2de8ef1d74767338f9a3bd6850740682852f5395c16d64 Oct 30 23:19:29.183907 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.183874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:29.186830 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.186775 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:29.202206 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.201545 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:29.443447 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.443234 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:29.507314 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.506650 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586c8ddf94-s29tc"] Oct 30 23:19:29.588233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.588146 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:19:29.593370 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.593349 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.598047 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.598021 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 23:19:29.598130 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.598058 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 23:19:29.598130 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.598128 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 23:19:29.598506 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.598490 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 23:19:29.598602 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.598543 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 23:19:29.598602 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.598585 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 23:19:29.598717 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.598610 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wczpt\"" Oct 30 23:19:29.598717 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.598620 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 23:19:29.598717 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.598615 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 23:19:29.598717 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.598635 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 23:19:29.609556 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.609479 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:19:29.692521 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.692482 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.692650 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.692562 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.692717 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.692644 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-web-config\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.692717 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.692681 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwsvr\" (UniqueName: \"kubernetes.io/projected/5973a913-5fc2-45b7-affa-4184ee2da449-kube-api-access-vwsvr\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.692795 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.692719 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5973a913-5fc2-45b7-affa-4184ee2da449-config-out\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.692795 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.692749 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5973a913-5fc2-45b7-affa-4184ee2da449-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.692859 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.692819 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.692893 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.692861 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.692933 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.692890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5973a913-5fc2-45b7-affa-4184ee2da449-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.692933 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.692928 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-config-volume\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.692996 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.692972 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5973a913-5fc2-45b7-affa-4184ee2da449-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.692996 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.692988 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.693056 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.693003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5973a913-5fc2-45b7-affa-4184ee2da449-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.732590 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.732528 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4"] Oct 30 23:19:29.736582 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:29.736552 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd5ef23_9d7a_44c2_b0fe_653a2cba55e0.slice/crio-1fbc8f4e25b798803a0f032a61746747020e245401077974df4d9ee1441460f6 WatchSource:0}: Error finding container 1fbc8f4e25b798803a0f032a61746747020e245401077974df4d9ee1441460f6: Status 404 returned error can't find the container with id 1fbc8f4e25b798803a0f032a61746747020e245401077974df4d9ee1441460f6 Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.793987 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794032 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5973a913-5fc2-45b7-affa-4184ee2da449-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794066 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-config-volume\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794107 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5973a913-5fc2-45b7-affa-4184ee2da449-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794134 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794155 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5973a913-5fc2-45b7-affa-4184ee2da449-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794193 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794215 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-web-config\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794271 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwsvr\" (UniqueName: \"kubernetes.io/projected/5973a913-5fc2-45b7-affa-4184ee2da449-kube-api-access-vwsvr\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794304 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5973a913-5fc2-45b7-affa-4184ee2da449-config-out\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5973a913-5fc2-45b7-affa-4184ee2da449-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.794773 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.794751 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5973a913-5fc2-45b7-affa-4184ee2da449-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.795559 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.795502 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5973a913-5fc2-45b7-affa-4184ee2da449-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.795961 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.795927 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5973a913-5fc2-45b7-affa-4184ee2da449-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.800091 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.800045 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.800504 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.800437 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5973a913-5fc2-45b7-affa-4184ee2da449-config-out\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.801526 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.801047 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-config-volume\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.801670 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.801455 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.802757 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.802735 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5973a913-5fc2-45b7-affa-4184ee2da449-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.802874 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.802852 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.803519 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.803418 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.803958 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.803917 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.809704 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.809680 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwsvr\" (UniqueName: \"kubernetes.io/projected/5973a913-5fc2-45b7-affa-4184ee2da449-kube-api-access-vwsvr\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.819087 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.819041 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-web-config\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:29.848996 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.848917 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" event={"ID":"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0","Type":"ContainerStarted","Data":"1fbc8f4e25b798803a0f032a61746747020e245401077974df4d9ee1441460f6"} Oct 30 23:19:29.851396 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.851362 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h4hmx" event={"ID":"cb76bb5e-c0a6-4a38-8732-40193dd39cac","Type":"ContainerStarted","Data":"551cd33b8f2ec26841bda929238bd81463fe74bbbf3239164f5f2ac5940e9f17"} Oct 30 23:19:29.851542 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.851420 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h4hmx" event={"ID":"cb76bb5e-c0a6-4a38-8732-40193dd39cac","Type":"ContainerStarted","Data":"b84dbc8ba60c972f7f36639dd47c99c775912338f9eddcf4cd69667e6e13e601"} Oct 30 23:19:29.851625 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.851608 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:29.853676 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.853648 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586c8ddf94-s29tc" event={"ID":"b5349bf3-d267-4d03-8f45-60e58c5f881e","Type":"ContainerStarted","Data":"e0a005ab3d10d39c642af76fa383d89f209ea00f95db3ba5eee5a428839c7409"} Oct 30 23:19:29.855540 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.855515 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hq9sq" event={"ID":"a0371e4c-ff4b-4867-98a1-bb598ab03845","Type":"ContainerStarted","Data":"ecbb743dd6362c36741ee41c3b8ae2d115ec350e24a5567b3b67d3251a24ffd3"} Oct 30 23:19:29.858395 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.858281 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" event={"ID":"68aefc1f-0a32-4722-a494-054f46f44b5b","Type":"ContainerStarted","Data":"2ab8df9e978f0ef30c90db0d35357c2c8d5b4d7b1a5506b143c4e49362f7b885"} Oct 30 23:19:29.858395 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.858330 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" event={"ID":"68aefc1f-0a32-4722-a494-054f46f44b5b","Type":"ContainerStarted","Data":"3fa81d1ed64b3aeff512d2f8a834327f39ce9c77b2f8038f305033fe0d285e1a"} Oct 30 23:19:29.858395 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.858358 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" event={"ID":"68aefc1f-0a32-4722-a494-054f46f44b5b","Type":"ContainerStarted","Data":"150395e7507b6aff8a2de8ef1d74767338f9a3bd6850740682852f5395c16d64"} Oct 30 23:19:29.922640 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:29.922593 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:30.250172 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.249255 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-h4hmx" podStartSLOduration=2.125521372 podStartE2EDuration="4.24923322s" totalImagesPullingTime="2.123711848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:29.873873772 +0000 UTC m=+53.826981098" watchObservedRunningTime="2026-10-30 23:19:30.24923322 +0000 UTC m=+54.202340547" Oct 30 23:19:30.250172 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.250056 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:19:30.268721 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:30.268683 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5973a913_5fc2_45b7_affa_4184ee2da449.slice/crio-35b812a727e8b930440d3ccc2bbffc9ce63c1a660e4cd7764db966991701d532 WatchSource:0}: Error finding container 35b812a727e8b930440d3ccc2bbffc9ce63c1a660e4cd7764db966991701d532: Status 404 returned error can't find the container with id 35b812a727e8b930440d3ccc2bbffc9ce63c1a660e4cd7764db966991701d532 Oct 30 23:19:30.581070 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.581037 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2"] Oct 30 23:19:30.584920 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.584896 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.595502 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.595475 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 23:19:30.595878 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.595559 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 23:19:30.595878 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.595561 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 23:19:30.596380 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.596066 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-anck8g3riltac\"" Oct 30 23:19:30.596380 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.596153 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 23:19:30.596380 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.596195 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 23:19:30.596380 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.596237 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8gxz9\"" Oct 30 23:19:30.599072 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.599049 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2"] Oct 30 23:19:30.704248 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.703732 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.704248 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.703801 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.704248 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.703825 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-grpc-tls\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.704248 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.703855 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.704248 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.703885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5h84\" (UniqueName: \"kubernetes.io/projected/d189925d-3211-43c6-8bf0-e9aca119e00e-kube-api-access-t5h84\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.704248 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.703910 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d189925d-3211-43c6-8bf0-e9aca119e00e-metrics-client-ca\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.704248 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.703948 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-tls\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.704248 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.703989 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.704248 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.704027 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d189925d-3211-43c6-8bf0-e9aca119e00e-tmp\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.806011 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.804887 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.806011 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.804950 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.806011 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.804985 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-grpc-tls\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.806011 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.805015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.806011 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.805044 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5h84\" (UniqueName: \"kubernetes.io/projected/d189925d-3211-43c6-8bf0-e9aca119e00e-kube-api-access-t5h84\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.806011 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.805070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d189925d-3211-43c6-8bf0-e9aca119e00e-metrics-client-ca\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.806011 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.805108 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-tls\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.806011 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.805149 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.806011 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.805193 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d189925d-3211-43c6-8bf0-e9aca119e00e-tmp\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.806011 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.805641 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d189925d-3211-43c6-8bf0-e9aca119e00e-tmp\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.806578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.806134 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d189925d-3211-43c6-8bf0-e9aca119e00e-metrics-client-ca\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.810087 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.810032 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.812938 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.812887 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.817237 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.817211 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-tls\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.817605 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.817583 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.817869 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.817814 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.818419 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.818396 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5h84\" (UniqueName: \"kubernetes.io/projected/d189925d-3211-43c6-8bf0-e9aca119e00e-kube-api-access-t5h84\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.822132 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.822111 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-grpc-tls\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.863334 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.863258 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5973a913-5fc2-45b7-affa-4184ee2da449","Type":"ContainerStarted","Data":"35b812a727e8b930440d3ccc2bbffc9ce63c1a660e4cd7764db966991701d532"} Oct 30 23:19:30.869693 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.869659 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vq6t" event={"ID":"d54e0641-5f0d-41bd-8b1b-900463a24cfa","Type":"ContainerStarted","Data":"4b2141d9ec2ac8acb02186b5add60a628ba10bb95e65a20e1a2b415ad3b09bbf"} Oct 30 23:19:30.874236 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.874180 2581 generic.go:356] "Generic (PLEG): container finished" podID="a0371e4c-ff4b-4867-98a1-bb598ab03845" containerID="fd2d100fea6d36a632a73af0e0a1b200eb7df303efc76914a26439f9b9c1f58d" exitCode=0 Oct 30 23:19:30.874375 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.874257 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hq9sq" event={"ID":"a0371e4c-ff4b-4867-98a1-bb598ab03845","Type":"ContainerDied","Data":"fd2d100fea6d36a632a73af0e0a1b200eb7df303efc76914a26439f9b9c1f58d"} Oct 30 23:19:30.881282 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.881156 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" event={"ID":"68aefc1f-0a32-4722-a494-054f46f44b5b","Type":"ContainerStarted","Data":"629d716142f7abe8bd01df06837be5119c1477f583e1e98370c4a9c4d16e0ed9"} Oct 30 23:19:30.894456 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.894414 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7vq6t" podStartSLOduration=2.141132207 podStartE2EDuration="4.894401359s" totalImagesPullingTime="2.753269152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:19:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:30.893594613 +0000 UTC m=+54.846701940" watchObservedRunningTime="2026-10-30 23:19:30.894401359 +0000 UTC m=+54.847508682" Oct 30 23:19:30.916789 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.916761 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:30.938858 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:30.937860 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" podStartSLOduration=1.438329069 podStartE2EDuration="2.937841949s" totalImagesPullingTime="1.49951288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:30.937377158 +0000 UTC m=+54.890484480" watchObservedRunningTime="2026-10-30 23:19:30.937841949 +0000 UTC m=+54.890949277" Oct 30 23:19:31.221391 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:31.219891 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2"] Oct 30 23:19:31.223301 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:31.223253 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd189925d_3211_43c6_8bf0_e9aca119e00e.slice/crio-52b8ae133b36f50afb97f5e02233ee782fed5ca37c1576a45b50ef098165d960 WatchSource:0}: Error finding container 52b8ae133b36f50afb97f5e02233ee782fed5ca37c1576a45b50ef098165d960: Status 404 returned error can't find the container with id 52b8ae133b36f50afb97f5e02233ee782fed5ca37c1576a45b50ef098165d960 Oct 30 23:19:31.887530 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:31.887098 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hq9sq" event={"ID":"a0371e4c-ff4b-4867-98a1-bb598ab03845","Type":"ContainerStarted","Data":"a3e9147b2dcd6846dea65b83a42f529c57f4583a709db5d5c0cce6eb2ad54428"} Oct 30 23:19:31.887530 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:31.887162 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hq9sq" event={"ID":"a0371e4c-ff4b-4867-98a1-bb598ab03845","Type":"ContainerStarted","Data":"84497a98ed4b9a27dedf122fce823a93c2effb62c0fe5ae11fcd759fa4241048"} Oct 30 23:19:31.888748 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:31.888722 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" event={"ID":"d189925d-3211-43c6-8bf0-e9aca119e00e","Type":"ContainerStarted","Data":"52b8ae133b36f50afb97f5e02233ee782fed5ca37c1576a45b50ef098165d960"} Oct 30 23:19:31.892347 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:31.892296 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" event={"ID":"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0","Type":"ContainerStarted","Data":"0056a138b785fe0674b9cd03fcd70d75e42a8cd8133c5966c6b0f465c16219b5"} Oct 30 23:19:31.892523 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:31.892397 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" event={"ID":"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0","Type":"ContainerStarted","Data":"ad59e2df0bdb2d4758e155d71bd72c511843a7b85069ca217597f42daaaf82b8"} Oct 30 23:19:31.895311 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:31.895287 2581 generic.go:356] "Generic (PLEG): container finished" podID="5973a913-5fc2-45b7-affa-4184ee2da449" containerID="42e30929878643e9f9096f8cba26a4314c4d0d72f4a3c95105b953f6c19e89b0" exitCode=0 Oct 30 23:19:31.895408 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:31.895377 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5973a913-5fc2-45b7-affa-4184ee2da449","Type":"ContainerDied","Data":"42e30929878643e9f9096f8cba26a4314c4d0d72f4a3c95105b953f6c19e89b0"} Oct 30 23:19:31.898455 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:31.898435 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tcb2" event={"ID":"284dba08-fe12-4942-a5fd-6c1e2f979a17","Type":"ContainerStarted","Data":"9e759910c30ab745ff8643ceeb84e8babc1d96d0b43d70fa2f2b370d82d30bbc"} Oct 30 23:19:31.913328 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:31.912533 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hq9sq" podStartSLOduration=2.39267466 podStartE2EDuration="3.912518292s" totalImagesPullingTime="1.423208957s" totalInitContainerRuntime="96.634675ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:31.912157018 +0000 UTC m=+55.865264346" watchObservedRunningTime="2026-10-30 23:19:31.912518292 +0000 UTC m=+55.865625619" Oct 30 23:19:31.966598 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:31.965880 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9tcb2" podStartSLOduration=1.664309866 podStartE2EDuration="5.965866761s" totalImagesPullingTime="4.301556895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:31.965495003 +0000 UTC m=+55.918602325" watchObservedRunningTime="2026-10-30 23:19:31.965866761 +0000 UTC m=+55.918974083" Oct 30 23:19:32.882791 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.882754 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5946dc94c7-djbgv"] Oct 30 23:19:32.888622 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.887608 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:32.892577 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.892495 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 23:19:32.893072 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.893053 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 23:19:32.906843 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.906813 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5946dc94c7-djbgv"] Oct 30 23:19:32.918874 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.918844 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" event={"ID":"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0","Type":"ContainerStarted","Data":"99990baffaeb2f57833911d08e99f72d8c13a6b794ac6890d67ee2b265d85ed7"} Oct 30 23:19:32.926549 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.926521 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 23:19:32.926824 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.926807 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1j1fi1cun0mic\"" Oct 30 23:19:32.927014 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.927000 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 23:19:32.927167 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.927153 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ww7vc\"" Oct 30 23:19:32.930721 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.930691 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-secret-metrics-server-client-certs\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:32.930811 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.930772 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed360b42-1a02-4474-9bf1-2117d1b438f1-metrics-server-audit-profiles\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:32.930886 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.930834 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z57gl\" (UniqueName: \"kubernetes.io/projected/ed360b42-1a02-4474-9bf1-2117d1b438f1-kube-api-access-z57gl\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:32.930886 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.930869 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-secret-metrics-server-tls\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:32.930985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.930896 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed360b42-1a02-4474-9bf1-2117d1b438f1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:32.930985 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.930956 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-client-ca-bundle\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:32.931114 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.931013 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed360b42-1a02-4474-9bf1-2117d1b438f1-audit-log\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:32.942314 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:32.942198 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" podStartSLOduration=3.063631984 podStartE2EDuration="4.942184614s" totalImagesPullingTime="1.87855263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:32.941052365 +0000 UTC m=+56.894159693" watchObservedRunningTime="2026-10-30 23:19:32.942184614 +0000 UTC m=+56.895291939" Oct 30 23:19:33.032358 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.032314 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-secret-metrics-server-client-certs\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.032553 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.032432 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed360b42-1a02-4474-9bf1-2117d1b438f1-metrics-server-audit-profiles\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.032613 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.032570 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z57gl\" (UniqueName: \"kubernetes.io/projected/ed360b42-1a02-4474-9bf1-2117d1b438f1-kube-api-access-z57gl\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.032669 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.032627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-secret-metrics-server-tls\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.032669 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.032661 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed360b42-1a02-4474-9bf1-2117d1b438f1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.032765 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.032738 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-client-ca-bundle\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.032867 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.032848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed360b42-1a02-4474-9bf1-2117d1b438f1-audit-log\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.033274 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.033252 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed360b42-1a02-4474-9bf1-2117d1b438f1-audit-log\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.035659 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.034992 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed360b42-1a02-4474-9bf1-2117d1b438f1-metrics-server-audit-profiles\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.035972 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.035949 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed360b42-1a02-4474-9bf1-2117d1b438f1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.036555 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.036505 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-secret-metrics-server-client-certs\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.036662 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.036639 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-secret-metrics-server-tls\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.037523 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.037502 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-client-ca-bundle\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.048066 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.048038 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z57gl\" (UniqueName: \"kubernetes.io/projected/ed360b42-1a02-4474-9bf1-2117d1b438f1-kube-api-access-z57gl\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.216490 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.216387 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:33.267640 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.267586 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-g86k2"] Oct 30 23:19:33.272358 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.272330 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:33.278984 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.278958 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lzqxs\"" Oct 30 23:19:33.278984 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.278990 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 23:19:33.281361 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.281340 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-g86k2"] Oct 30 23:19:33.338054 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.337901 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvkv\" (UniqueName: \"kubernetes.io/projected/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-kube-api-access-swvkv\") pod \"monitoring-plugin-77f4f76594-g86k2\" (UID: \"9c20fb11-2dcf-445e-96cd-4d97045a2ab9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:33.338054 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.337959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-g86k2\" (UID: \"9c20fb11-2dcf-445e-96cd-4d97045a2ab9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:33.438689 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.438651 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvkv\" (UniqueName: \"kubernetes.io/projected/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-kube-api-access-swvkv\") pod \"monitoring-plugin-77f4f76594-g86k2\" (UID: \"9c20fb11-2dcf-445e-96cd-4d97045a2ab9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:33.438848 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.438732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-g86k2\" (UID: \"9c20fb11-2dcf-445e-96cd-4d97045a2ab9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:33.438942 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:33.438924 2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Oct 30 23:19:33.439011 ip-10-0-137-81 kubenswrapper[2581]: E1030 23:19:33.439000 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-monitoring-plugin-cert podName:9c20fb11-2dcf-445e-96cd-4d97045a2ab9 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:33.938981402 +0000 UTC m=+57.892088711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-monitoring-plugin-cert") pod "monitoring-plugin-77f4f76594-g86k2" (UID: "9c20fb11-2dcf-445e-96cd-4d97045a2ab9") : secret "monitoring-plugin-cert" not found Oct 30 23:19:33.455684 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.455652 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvkv\" (UniqueName: \"kubernetes.io/projected/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-kube-api-access-swvkv\") pod \"monitoring-plugin-77f4f76594-g86k2\" (UID: \"9c20fb11-2dcf-445e-96cd-4d97045a2ab9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:33.548419 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.548334 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5946dc94c7-djbgv"] Oct 30 23:19:33.550827 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:33.550799 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded360b42_1a02_4474_9bf1_2117d1b438f1.slice/crio-d96a79c079e48113babe4f1525deef623e4cba5f911392bfe10010af64b455a7 WatchSource:0}: Error finding container d96a79c079e48113babe4f1525deef623e4cba5f911392bfe10010af64b455a7: Status 404 returned error can't find the container with id d96a79c079e48113babe4f1525deef623e4cba5f911392bfe10010af64b455a7 Oct 30 23:19:33.706038 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.706007 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-976bbf6cb-lk9st"] Oct 30 23:19:33.709662 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.709644 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.713837 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.713785 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 23:19:33.713837 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.713792 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 23:19:33.713837 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.713808 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 23:19:33.713837 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.713822 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rpncv\"" Oct 30 23:19:33.714877 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.714856 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 23:19:33.715233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.715184 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 23:19:33.719981 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.719959 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 23:19:33.726444 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.726420 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-976bbf6cb-lk9st"] Oct 30 23:19:33.741502 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.741431 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-metrics-client-ca\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.741502 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.741497 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-serving-certs-ca-bundle\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.741633 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.741549 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-federate-client-tls\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.741668 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.741637 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.741698 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.741676 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-secret-telemeter-client\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.741727 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.741706 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-telemeter-trusted-ca-bundle\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.741758 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.741752 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-telemeter-client-tls\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.741787 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.741774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvw4\" (UniqueName: \"kubernetes.io/projected/d0b4de16-5f57-44fe-a273-a77ebd48d050-kube-api-access-pmvw4\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.804413 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.803930 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:33.843101 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.843072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvw4\" (UniqueName: \"kubernetes.io/projected/d0b4de16-5f57-44fe-a273-a77ebd48d050-kube-api-access-pmvw4\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.843180 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.843136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-metrics-client-ca\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.843231 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.843216 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-serving-certs-ca-bundle\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.843262 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.843245 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-federate-client-tls\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.843310 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.843297 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.843351 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.843341 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-secret-telemeter-client\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.843394 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.843376 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-telemeter-trusted-ca-bundle\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.843452 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.843412 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-telemeter-client-tls\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.844541 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.844513 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-serving-certs-ca-bundle\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.847532 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.846387 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-metrics-client-ca\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.848834 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.848157 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-telemeter-client-tls\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.856812 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.853198 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-telemeter-trusted-ca-bundle\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.856812 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.853257 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-federate-client-tls\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.856812 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.853716 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.863909 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.863888 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-secret-telemeter-client\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.869323 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.869305 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvw4\" (UniqueName: \"kubernetes.io/projected/d0b4de16-5f57-44fe-a273-a77ebd48d050-kube-api-access-pmvw4\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:33.923340 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.923311 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586c8ddf94-s29tc" event={"ID":"b5349bf3-d267-4d03-8f45-60e58c5f881e","Type":"ContainerStarted","Data":"8c67e66da002223816af66292992c69acafb3705be263bd61e4656a7ad48f4a3"} Oct 30 23:19:33.926248 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.926226 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" event={"ID":"d189925d-3211-43c6-8bf0-e9aca119e00e","Type":"ContainerStarted","Data":"e3b93462c6bfe7c40eb3cd7d9d3104119c554709ff072dae0cfec9214ea60d02"} Oct 30 23:19:33.926870 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.926847 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" event={"ID":"d189925d-3211-43c6-8bf0-e9aca119e00e","Type":"ContainerStarted","Data":"d7812543b66b3501d7f05a5827e1382e0b92262823206ba7f3faccc86a8d507a"} Oct 30 23:19:33.930649 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.930628 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5973a913-5fc2-45b7-affa-4184ee2da449","Type":"ContainerStarted","Data":"45c5c5cca204fbe475e937e205f4c7c46fd6fb5f19278409281d4667de07f423"} Oct 30 23:19:33.930724 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.930688 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5973a913-5fc2-45b7-affa-4184ee2da449","Type":"ContainerStarted","Data":"f244a6b5b017f3f0e5c4a9e45c33c906f5dfd089f68278ee06a66c9f4106d86f"} Oct 30 23:19:33.931821 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.931801 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" event={"ID":"ed360b42-1a02-4474-9bf1-2117d1b438f1","Type":"ContainerStarted","Data":"d96a79c079e48113babe4f1525deef623e4cba5f911392bfe10010af64b455a7"} Oct 30 23:19:33.945014 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.944983 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-g86k2\" (UID: \"9c20fb11-2dcf-445e-96cd-4d97045a2ab9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:33.947518 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:33.947499 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-g86k2\" (UID: \"9c20fb11-2dcf-445e-96cd-4d97045a2ab9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:34.038916 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.038892 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:34.189688 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.189652 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:34.321999 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.321894 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-586c8ddf94-s29tc" podStartSLOduration=2.347845473 podStartE2EDuration="6.32187409s" totalImagesPullingTime="3.974028617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:33.949790631 +0000 UTC m=+57.902897958" watchObservedRunningTime="2026-10-30 23:19:34.32187409 +0000 UTC m=+58.274981418" Oct 30 23:19:34.322613 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.322594 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-976bbf6cb-lk9st"] Oct 30 23:19:34.325733 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:34.325707 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b4de16_5f57_44fe_a273_a77ebd48d050.slice/crio-8573761214d01e14398d2b3c6c7915d7f677e2e65804378f31aa4072d183cf23 WatchSource:0}: Error finding container 8573761214d01e14398d2b3c6c7915d7f677e2e65804378f31aa4072d183cf23: Status 404 returned error can't find the container with id 8573761214d01e14398d2b3c6c7915d7f677e2e65804378f31aa4072d183cf23 Oct 30 23:19:34.463072 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.463040 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-g86k2"] Oct 30 23:19:34.467013 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:34.466971 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c20fb11_2dcf_445e_96cd_4d97045a2ab9.slice/crio-b222d723beb7f4f324e9e01fa101c41e1ed6a3d02dc72a4a3c1d68d0ad39054f WatchSource:0}: Error finding container b222d723beb7f4f324e9e01fa101c41e1ed6a3d02dc72a4a3c1d68d0ad39054f: Status 404 returned error can't find the container with id b222d723beb7f4f324e9e01fa101c41e1ed6a3d02dc72a4a3c1d68d0ad39054f Oct 30 23:19:34.751194 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.751163 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:19:34.759136 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.759010 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.766527 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xmrvg\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.766580 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.766617 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.766644 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.766713 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.767009 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7fh9lroec4t9q\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.767078 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.767141 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.767223 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.767286 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.767348 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.767422 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.767477 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 23:19:34.767578 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.767551 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 23:19:34.776692 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.776663 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:19:34.777212 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.777188 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 23:19:34.789484 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.786043 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 23:19:34.853579 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853537 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853746 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853746 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853640 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853746 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853663 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjz76\" (UniqueName: \"kubernetes.io/projected/da3a65c4-b342-42d7-aad0-29e63b90125a-kube-api-access-zjz76\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853746 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853689 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853746 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853999 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853749 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853999 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853817 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853999 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853840 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da3a65c4-b342-42d7-aad0-29e63b90125a-config-out\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853999 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853858 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853999 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853999 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853909 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853999 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853929 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853999 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853958 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.853999 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.853981 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-web-config\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.854432 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.854008 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-config\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.854432 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.854062 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da3a65c4-b342-42d7-aad0-29e63b90125a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.854432 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.854088 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.854432 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.854127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.854432 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.854149 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.947827 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.947692 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" event={"ID":"d189925d-3211-43c6-8bf0-e9aca119e00e","Type":"ContainerStarted","Data":"9036863cbd6b1bbcd5a89277e03d24189d09f8ac12ac1e8d1d3371de99c021fe"} Oct 30 23:19:34.953364 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.953331 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5973a913-5fc2-45b7-affa-4184ee2da449","Type":"ContainerStarted","Data":"81ece4272491b3a2fdf6f28df1c4bc9c8bb86224765b33322607337dab0b8a47"} Oct 30 23:19:34.953517 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.953417 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5973a913-5fc2-45b7-affa-4184ee2da449","Type":"ContainerStarted","Data":"9269cb01e57df068af28442030b790e9dd1c05bf849bd8257dfd4d5621d7fa8d"} Oct 30 23:19:34.953517 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.953480 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5973a913-5fc2-45b7-affa-4184ee2da449","Type":"ContainerStarted","Data":"30c60a3e82ed0af649ba722ecbfa1349c2021f742cb32c86c6dfb0ff15c339e5"} Oct 30 23:19:34.954703 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.954680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.954829 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.954709 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-web-config\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.954829 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.954718 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" event={"ID":"9c20fb11-2dcf-445e-96cd-4d97045a2ab9","Type":"ContainerStarted","Data":"b222d723beb7f4f324e9e01fa101c41e1ed6a3d02dc72a4a3c1d68d0ad39054f"} Oct 30 23:19:34.954829 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.954727 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-config\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.954829 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.954799 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da3a65c4-b342-42d7-aad0-29e63b90125a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955034 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.954862 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955034 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.954891 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955034 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.954940 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955034 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955017 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955058 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955125 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjz76\" (UniqueName: \"kubernetes.io/projected/da3a65c4-b342-42d7-aad0-29e63b90125a-kube-api-access-zjz76\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955172 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955215 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955451 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955451 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955359 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955451 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955383 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da3a65c4-b342-42d7-aad0-29e63b90125a-config-out\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955451 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955399 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955451 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955421 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955451 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955440 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.955758 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.955486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.957699 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.957452 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.957699 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.957483 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.957846 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.957818 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.958402 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.958373 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" event={"ID":"d0b4de16-5f57-44fe-a273-a77ebd48d050","Type":"ContainerStarted","Data":"8573761214d01e14398d2b3c6c7915d7f677e2e65804378f31aa4072d183cf23"} Oct 30 23:19:34.959139 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.959113 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.959539 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.959516 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.960166 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.960140 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.960415 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.960392 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.961158 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.961137 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.961807 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.961755 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-web-config\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.961937 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.961794 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.962054 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.961816 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.962800 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.962744 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da3a65c4-b342-42d7-aad0-29e63b90125a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.962800 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.962764 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.962921 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.962886 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.963022 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.962999 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.963115 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.963094 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da3a65c4-b342-42d7-aad0-29e63b90125a-config-out\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.964609 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.964569 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.965042 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.965024 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.968979 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.968961 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-config\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:34.972106 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:34.972084 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjz76\" (UniqueName: \"kubernetes.io/projected/da3a65c4-b342-42d7-aad0-29e63b90125a-kube-api-access-zjz76\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:35.108844 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:35.108647 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:35.416080 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:35.415594 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:19:35.424798 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:35.424149 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3a65c4_b342_42d7_aad0_29e63b90125a.slice/crio-1e17150f06114c7bf69037957484c81ab98d5e99115839018db8c95455d2da51 WatchSource:0}: Error finding container 1e17150f06114c7bf69037957484c81ab98d5e99115839018db8c95455d2da51: Status 404 returned error can't find the container with id 1e17150f06114c7bf69037957484c81ab98d5e99115839018db8c95455d2da51 Oct 30 23:19:35.567569 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:35.567538 2581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586c8ddf94-s29tc"] Oct 30 23:19:35.965402 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:35.965318 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5973a913-5fc2-45b7-affa-4184ee2da449","Type":"ContainerStarted","Data":"f0c1444c2664345703cb0f31180fb50c9d28e6d66dced8535de2b9f9bd97e55a"} Oct 30 23:19:35.968013 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:35.967804 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" event={"ID":"ed360b42-1a02-4474-9bf1-2117d1b438f1","Type":"ContainerStarted","Data":"d6c071d29384152b592df801487206c5ca8e68ec1e11a4f296680f8452dffa8d"} Oct 30 23:19:35.969777 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:35.969750 2581 generic.go:356] "Generic (PLEG): container finished" podID="da3a65c4-b342-42d7-aad0-29e63b90125a" containerID="fe5d7d437a54cfb0d94e9119385dc2d4b5ef9244d9e3a1dd605ec3c81565a814" exitCode=0 Oct 30 23:19:35.969902 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:35.969886 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da3a65c4-b342-42d7-aad0-29e63b90125a","Type":"ContainerDied","Data":"fe5d7d437a54cfb0d94e9119385dc2d4b5ef9244d9e3a1dd605ec3c81565a814"} Oct 30 23:19:35.969971 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:35.969954 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da3a65c4-b342-42d7-aad0-29e63b90125a","Type":"ContainerStarted","Data":"1e17150f06114c7bf69037957484c81ab98d5e99115839018db8c95455d2da51"} Oct 30 23:19:35.976051 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:35.975901 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" event={"ID":"d189925d-3211-43c6-8bf0-e9aca119e00e","Type":"ContainerStarted","Data":"76fbc54a0d78965feb17cc7986fcc998a9ad4ae9a6567f07b4f2d48a603a2e77"} Oct 30 23:19:35.976051 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:35.975966 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" event={"ID":"d189925d-3211-43c6-8bf0-e9aca119e00e","Type":"ContainerStarted","Data":"d7655dfb5f65cf430abc37d2cbc6461f5ec5fcad728af47defdddb795a25cf61"} Oct 30 23:19:35.976051 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:35.976004 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" event={"ID":"d189925d-3211-43c6-8bf0-e9aca119e00e","Type":"ContainerStarted","Data":"cbc61004ec3fe452e979af762090b79990fe5f564d6302c2962239b03a948776"} Oct 30 23:19:36.001638 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:36.001581 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.599196686 podStartE2EDuration="7.001565819s" totalImagesPullingTime="4.3765689s" totalInitContainerRuntime="25.800233ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:35.999788024 +0000 UTC m=+59.952895366" watchObservedRunningTime="2026-10-30 23:19:36.001565819 +0000 UTC m=+59.954673163" Oct 30 23:19:36.088695 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:36.088634 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podStartSLOduration=2.327447433 podStartE2EDuration="6.088613158s" totalImagesPullingTime="3.761165725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:36.08766104 +0000 UTC m=+60.040768367" watchObservedRunningTime="2026-10-30 23:19:36.088613158 +0000 UTC m=+60.041720485" Oct 30 23:19:36.089629 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:36.089590 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" podStartSLOduration=1.9510522620000001 podStartE2EDuration="4.089577461s" totalImagesPullingTime="2.138525199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:36.058083581 +0000 UTC m=+60.011190911" watchObservedRunningTime="2026-10-30 23:19:36.089577461 +0000 UTC m=+60.042684788" Oct 30 23:19:36.981775 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:36.981690 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" event={"ID":"9c20fb11-2dcf-445e-96cd-4d97045a2ab9","Type":"ContainerStarted","Data":"bc17dbaa645ad94f1a23916b06056cd1b4da647110001578c3e519cbf2e94211"} Oct 30 23:19:36.982239 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:36.982211 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:36.984675 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:36.984652 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" event={"ID":"d0b4de16-5f57-44fe-a273-a77ebd48d050","Type":"ContainerStarted","Data":"c80799671a24b1a4e45cd807f32e2e978eb19bfa087aeddecd705ec1f648f05f"} Oct 30 23:19:36.984766 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:36.984699 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" event={"ID":"d0b4de16-5f57-44fe-a273-a77ebd48d050","Type":"ContainerStarted","Data":"39ccb28a5fada6bc54d19e3e85547fdc68484727c2d565934556fcfae2e44eb1"} Oct 30 23:19:36.984766 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:36.984721 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" event={"ID":"d0b4de16-5f57-44fe-a273-a77ebd48d050","Type":"ContainerStarted","Data":"fd6c316eb11a77e120bbef1fdf9e7bd1f9e3f122310f9decb4f7859ccb117298"} Oct 30 23:19:36.985879 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:36.985855 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:36.987831 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:36.987807 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:37.002845 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:37.002797 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" podStartSLOduration=2.10647713 podStartE2EDuration="4.002784552s" totalImagesPullingTime="1.896307422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:37.001072195 +0000 UTC m=+60.954179541" watchObservedRunningTime="2026-10-30 23:19:37.002784552 +0000 UTC m=+60.955891893" Oct 30 23:19:37.027646 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:37.027596 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" podStartSLOduration=1.761152933 podStartE2EDuration="4.027581963s" totalImagesPullingTime="2.26642903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:37.026491551 +0000 UTC m=+60.979598878" watchObservedRunningTime="2026-10-30 23:19:37.027581963 +0000 UTC m=+60.980689478" Oct 30 23:19:39.202284 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:39.202259 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:39.882849 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:39.882816 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:40.005133 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:40.003831 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da3a65c4-b342-42d7-aad0-29e63b90125a","Type":"ContainerStarted","Data":"3889a010a867692f49f6b51de32f3f5959c8492dc376b0740c992baf43ccf5dd"} Oct 30 23:19:40.005133 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:40.003912 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da3a65c4-b342-42d7-aad0-29e63b90125a","Type":"ContainerStarted","Data":"b683aa5edb46585ce53dd8964e55670e5bd270917acb82fd57885873521fed63"} Oct 30 23:19:40.005133 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:40.003949 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da3a65c4-b342-42d7-aad0-29e63b90125a","Type":"ContainerStarted","Data":"fee2246b637c69b55b9f42513b685cc347891c1624f120ec2e76145e349594fa"} Oct 30 23:19:40.005133 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:40.003984 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da3a65c4-b342-42d7-aad0-29e63b90125a","Type":"ContainerStarted","Data":"09578c1e0c1b4b089236c069671639dec9ca3d0dde76d6ef0948e9952e4cbf0b"} Oct 30 23:19:40.005133 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:40.004018 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da3a65c4-b342-42d7-aad0-29e63b90125a","Type":"ContainerStarted","Data":"6db1cd167080df0b3c577a18815b815f7d8a7e881a3f4f7a4246e11f97bf176e"} Oct 30 23:19:40.005133 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:40.004052 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da3a65c4-b342-42d7-aad0-29e63b90125a","Type":"ContainerStarted","Data":"3de4ff0b7e207fc0a9db19691d7c58c5f4ad48074a8838530d1a6241e96ceca4"} Oct 30 23:19:40.053258 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:40.053197 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.945232128 podStartE2EDuration="6.053178229s" totalImagesPullingTime="3.082309779s" totalInitContainerRuntime="25.636322ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:40.050617647 +0000 UTC m=+64.003724973" watchObservedRunningTime="2026-10-30 23:19:40.053178229 +0000 UTC m=+64.006285556" Oct 30 23:19:40.110166 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:40.110112 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:42.237421 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.236980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:42.241822 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.241772 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:19:42.250453 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.250425 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:42.263484 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.263448 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-djdr2\"" Oct 30 23:19:42.271545 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.271521 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:42.340024 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.338544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mqf\" (UniqueName: \"kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf\") pod \"network-check-target-rhm2r\" (UID: \"a10c4ed2-0361-47e1-8810-8a710b2be42d\") " pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:42.340024 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.338668 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:42.341894 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.341796 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:19:42.342225 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.342017 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:19:42.352694 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.352661 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:19:42.353028 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.352973 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:42.369819 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.369619 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8mqf\" (UniqueName: \"kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf\") pod \"network-check-target-rhm2r\" (UID: \"a10c4ed2-0361-47e1-8810-8a710b2be42d\") " pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:42.548580 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.548498 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2nz9z\"" Oct 30 23:19:42.555721 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.555696 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:42.555873 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.555783 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9tmfn\"" Oct 30 23:19:42.563021 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.563000 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:42.580981 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.580868 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jgq5s"] Oct 30 23:19:42.596428 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:42.596397 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cad31bd_cac3_44ca_bb2c_ca302c4e82e4.slice/crio-ec1f1f350e3372f1cfe61c32f39c5694a261ad716e546d94cc74a09fcef92a46 WatchSource:0}: Error finding container ec1f1f350e3372f1cfe61c32f39c5694a261ad716e546d94cc74a09fcef92a46: Status 404 returned error can't find the container with id ec1f1f350e3372f1cfe61c32f39c5694a261ad716e546d94cc74a09fcef92a46 Oct 30 23:19:42.854233 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.854085 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rhm2r"] Oct 30 23:19:42.856533 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:42.856506 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10c4ed2_0361_47e1_8810_8a710b2be42d.slice/crio-efd1d20e544865c8f96659a57ef6fc51bc0898ef703d31aef0bc83efac2b1cc2 WatchSource:0}: Error finding container efd1d20e544865c8f96659a57ef6fc51bc0898ef703d31aef0bc83efac2b1cc2: Status 404 returned error can't find the container with id efd1d20e544865c8f96659a57ef6fc51bc0898ef703d31aef0bc83efac2b1cc2 Oct 30 23:19:42.880697 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.880616 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6sfch"] Oct 30 23:19:42.885424 ip-10-0-137-81 kubenswrapper[2581]: W1030 23:19:42.885401 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef6a2af_7976_41e3_af4e_c7e0cb5e6b3f.slice/crio-340ebca13d78c6232db1b8859aa77400dd71bc6fbd7b1a6d92de1b53c58c4f02 WatchSource:0}: Error finding container 340ebca13d78c6232db1b8859aa77400dd71bc6fbd7b1a6d92de1b53c58c4f02: Status 404 returned error can't find the container with id 340ebca13d78c6232db1b8859aa77400dd71bc6fbd7b1a6d92de1b53c58c4f02 Oct 30 23:19:42.995238 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:42.995205 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:43.023627 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:43.023577 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6sfch" event={"ID":"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f","Type":"ContainerStarted","Data":"340ebca13d78c6232db1b8859aa77400dd71bc6fbd7b1a6d92de1b53c58c4f02"} Oct 30 23:19:43.026133 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:43.026089 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jgq5s" event={"ID":"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4","Type":"ContainerStarted","Data":"ec1f1f350e3372f1cfe61c32f39c5694a261ad716e546d94cc74a09fcef92a46"} Oct 30 23:19:43.028067 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:43.028033 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rhm2r" event={"ID":"a10c4ed2-0361-47e1-8810-8a710b2be42d","Type":"ContainerStarted","Data":"efd1d20e544865c8f96659a57ef6fc51bc0898ef703d31aef0bc83efac2b1cc2"} Oct 30 23:19:45.047046 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:45.047009 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jgq5s" event={"ID":"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4","Type":"ContainerStarted","Data":"63652f969f8f073a1f35826d829b67589913040b7b9cfc895caeecc2f5c40904"} Oct 30 23:19:45.047539 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:45.047074 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jgq5s" event={"ID":"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4","Type":"ContainerStarted","Data":"e3b8fef05972ff3c984d14e56ca62e3660863e57d649155088d33c82af361ee9"} Oct 30 23:19:45.068833 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:45.068789 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jgq5s" podStartSLOduration=66.65442742 podStartE2EDuration="1m8.068775149s" totalImagesPullingTime="1.414347729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:45.06630558 +0000 UTC m=+69.019412907" watchObservedRunningTime="2026-10-30 23:19:45.068775149 +0000 UTC m=+69.021882529" Oct 30 23:19:48.061962 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:48.061918 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rhm2r" event={"ID":"a10c4ed2-0361-47e1-8810-8a710b2be42d","Type":"ContainerStarted","Data":"9eb388bf1c18e542bae2f7005921b5ddb4a1c50bf5f22cbae995414a72f6023f"} Oct 30 23:19:48.062502 ip-10-0-137-81 kubenswrapper[2581]: I1030 23:19:48.062241 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:48.420300 ip-10-0-137-81 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 23:19:48.431858 ip-10-0-137-81 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 23:19:48.432028 ip-10-0-137-81 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 23:19:48.432077 ip-10-0-137-81 systemd[1]: kubelet.service: Consumed 4.909s CPU time. Oct 30 23:19:48.446092 ip-10-0-137-81 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:19:48.511733 ip-10-0-137-81 kubenswrapper[7570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:19:48.511733 ip-10-0-137-81 kubenswrapper[7570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:19:48.511733 ip-10-0-137-81 kubenswrapper[7570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:19:48.511733 ip-10-0-137-81 kubenswrapper[7570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:19:48.513815 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513728 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:19:48.513815 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513815 7570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513819 7570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513822 7570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513826 7570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513828 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513831 7570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513834 7570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513836 7570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513839 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513842 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513844 7570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513854 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513857 7570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513860 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513862 7570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513865 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513867 7570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513870 7570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513873 7570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513876 7570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:19:48.513884 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513878 7570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513882 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513884 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513887 7570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513890 7570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513892 7570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513895 7570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513898 7570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513901 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513903 7570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513905 7570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513908 7570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513910 7570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513913 7570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513916 7570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513918 7570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513920 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513923 7570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513925 7570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513928 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:19:48.514381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513930 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513932 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513936 7570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513938 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513941 7570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513944 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513947 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513949 7570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513952 7570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513954 7570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513958 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513960 7570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513963 7570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513966 7570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513968 7570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513971 7570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513974 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513976 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513979 7570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:19:48.514895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513981 7570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513984 7570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513986 7570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513989 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513991 7570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513993 7570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513996 7570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.513998 7570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514000 7570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514003 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514006 7570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514008 7570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514010 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514015 7570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514018 7570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514022 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514025 7570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514028 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514031 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:19:48.515419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514033 7570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514036 7570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514038 7570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514041 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514043 7570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514046 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514048 7570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514051 7570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514054 7570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514058 7570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514061 7570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514064 7570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514066 7570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514069 7570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514071 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514073 7570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514076 7570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514078 7570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514081 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:19:48.515913 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514083 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514086 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514088 7570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514091 7570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514094 7570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514098 7570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514100 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514103 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514867 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514875 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514878 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514882 7570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514884 7570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514887 7570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514891 7570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514895 7570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514898 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514901 7570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514904 7570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:19:48.516385 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514907 7570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514910 7570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514913 7570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514915 7570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514918 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514920 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514923 7570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514925 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514928 7570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514930 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514933 7570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514935 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514938 7570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514940 7570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514943 7570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514945 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514948 7570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514950 7570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514953 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514956 7570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:19:48.516882 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514958 7570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514961 7570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514964 7570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514966 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514969 7570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514972 7570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514975 7570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514978 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514980 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514983 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514986 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514988 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514990 7570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514994 7570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514997 7570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.514999 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515002 7570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515004 7570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515007 7570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515009 7570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:19:48.517408 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515012 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515014 7570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515017 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515020 7570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515022 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515027 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515030 7570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515039 7570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515041 7570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515044 7570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515048 7570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515051 7570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515055 7570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515057 7570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515060 7570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515063 7570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515065 7570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515068 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:19:48.517925 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515078 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515081 7570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515083 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515086 7570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515088 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515091 7570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515093 7570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515096 7570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515099 7570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515101 7570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515104 7570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515106 7570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515108 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515111 7570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515113 7570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515116 7570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515118 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515121 7570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515123 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515127 7570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:19:48.518372 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515129 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515132 7570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515135 7570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515137 7570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515139 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515142 7570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515145 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515148 7570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515151 7570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515153 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515155 7570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515158 7570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515160 7570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515168 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515171 7570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515173 7570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515176 7570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515696 7570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515703 7570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:19:48.518895 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515707 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515711 7570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515713 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515716 7570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515719 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515722 7570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515725 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515727 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515730 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515732 7570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515735 7570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515737 7570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515740 7570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515743 7570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515746 7570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515749 7570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515751 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515753 7570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515756 7570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:19:48.519386 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515758 7570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515761 7570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515763 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515765 7570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515768 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515770 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515773 7570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515776 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515778 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515781 7570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515783 7570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515786 7570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515788 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515791 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515794 7570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515796 7570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515799 7570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515801 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515804 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515806 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:19:48.519877 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515809 7570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515812 7570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515814 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515817 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515819 7570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515822 7570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515825 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515827 7570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515829 7570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515832 7570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515834 7570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515837 7570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515839 7570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515843 7570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515846 7570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515849 7570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515852 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515855 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515859 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:19:48.520381 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515861 7570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515864 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515867 7570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515869 7570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515873 7570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515876 7570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515879 7570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515882 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515884 7570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515887 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515889 7570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515891 7570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515894 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515896 7570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515899 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515901 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515904 7570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515906 7570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515909 7570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:19:48.520909 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515911 7570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515913 7570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515916 7570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515919 7570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515922 7570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515925 7570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515927 7570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515930 7570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515933 7570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515935 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515938 7570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515940 7570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515943 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515945 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515948 7570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515951 7570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515953 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515956 7570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515960 7570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515963 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:19:48.521373 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515966 7570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515968 7570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515971 7570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515973 7570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515975 7570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515978 7570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.515980 7570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516067 7570 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516080 7570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516087 7570 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516094 7570 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516100 7570 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516105 7570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516111 7570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516115 7570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516118 7570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516123 7570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516127 7570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516131 7570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516134 7570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516137 7570 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516140 7570 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516144 7570 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:19:48.521883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516147 7570 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516150 7570 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516154 7570 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516158 7570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516161 7570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516164 7570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516168 7570 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516172 7570 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516175 7570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516178 7570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516181 7570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516184 7570 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516187 7570 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516190 7570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516193 7570 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516195 7570 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516200 7570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516202 7570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516205 7570 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516208 7570 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516211 7570 flags.go:64] FLAG: --enable-server="true" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516214 7570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516218 7570 flags.go:64] FLAG: --event-burst="100" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516221 7570 flags.go:64] FLAG: --event-qps="50" Oct 30 23:19:48.522770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516224 7570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516228 7570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516231 7570 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516236 7570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516239 7570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516242 7570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516245 7570 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516248 7570 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516251 7570 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516254 7570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516257 7570 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516260 7570 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516263 7570 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516265 7570 flags.go:64] FLAG: --feature-gates="" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516270 7570 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516273 7570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516276 7570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516279 7570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516282 7570 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516285 7570 flags.go:64] FLAG: --help="false" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516288 7570 flags.go:64] FLAG: --hostname-override="ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516291 7570 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516294 7570 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:19:48.523548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516297 7570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516300 7570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516304 7570 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516306 7570 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516309 7570 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516312 7570 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516314 7570 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516317 7570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516320 7570 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516323 7570 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516327 7570 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516330 7570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516334 7570 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516337 7570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516340 7570 flags.go:64] FLAG: --lock-file="" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516343 7570 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516346 7570 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516349 7570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516354 7570 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516357 7570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516360 7570 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516363 7570 flags.go:64] FLAG: --logging-format="text" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516366 7570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516369 7570 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:19:48.524117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516372 7570 flags.go:64] FLAG: --manifest-url="" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516375 7570 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516379 7570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516382 7570 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516387 7570 flags.go:64] FLAG: --max-pods="110" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516390 7570 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516393 7570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516396 7570 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516399 7570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516401 7570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516404 7570 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516407 7570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516414 7570 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516417 7570 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516420 7570 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516423 7570 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516426 7570 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516428 7570 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516433 7570 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516436 7570 flags.go:64] FLAG: --port="10250" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516440 7570 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516443 7570 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d02ee9bc8e5b0505" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516446 7570 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516449 7570 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516452 7570 flags.go:64] FLAG: --register-node="true" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516454 7570 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:19:48.524727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516475 7570 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516478 7570 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516481 7570 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516484 7570 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516487 7570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516490 7570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516494 7570 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516497 7570 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516501 7570 flags.go:64] FLAG: --runonce="false" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516503 7570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516506 7570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516509 7570 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516512 7570 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516515 7570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516518 7570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516521 7570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516523 7570 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516526 7570 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516529 7570 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516532 7570 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516534 7570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516537 7570 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516540 7570 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516543 7570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516550 7570 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:19:48.525377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516552 7570 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516557 7570 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516562 7570 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516565 7570 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516567 7570 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516570 7570 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516573 7570 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516576 7570 flags.go:64] FLAG: --v="2" Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516580 7570 flags.go:64] FLAG: --version="false" Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516584 7570 flags.go:64] FLAG: --vmodule="" Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516588 7570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.516591 7570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516727 7570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516731 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516735 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516738 7570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516742 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516745 7570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516747 7570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516750 7570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516752 7570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516755 7570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516757 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:19:48.526004 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516760 7570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516762 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516765 7570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516767 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516770 7570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516772 7570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516775 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516777 7570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516782 7570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516785 7570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516789 7570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516791 7570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516794 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516796 7570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516799 7570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516801 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516804 7570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516806 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516809 7570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516811 7570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:19:48.526634 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516814 7570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516816 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516819 7570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516821 7570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516823 7570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516826 7570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516829 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516831 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516834 7570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516836 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516838 7570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516841 7570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516843 7570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516846 7570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516848 7570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516851 7570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516853 7570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516856 7570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516858 7570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516860 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:19:48.527187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516864 7570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516867 7570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516871 7570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516873 7570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516876 7570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516878 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516880 7570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516883 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516886 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516888 7570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516891 7570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516893 7570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516896 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516899 7570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516901 7570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516904 7570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516907 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516909 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516914 7570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:19:48.527867 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516918 7570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516921 7570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516924 7570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516927 7570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516929 7570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516932 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516934 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516937 7570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516939 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516941 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516944 7570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516947 7570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516950 7570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516954 7570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516957 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516961 7570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516963 7570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516966 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516968 7570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:19:48.528592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516971 7570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516973 7570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516976 7570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516978 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516980 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516983 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516985 7570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516988 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516991 7570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516993 7570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516996 7570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.516998 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517001 7570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517004 7570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517006 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517009 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:19:48.529067 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517011 7570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.517038 7570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517075 7570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517079 7570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517082 7570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517085 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517088 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517090 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517093 7570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517095 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517098 7570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517103 7570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517107 7570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517110 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517112 7570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:19:48.529559 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517115 7570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517117 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517120 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517122 7570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517125 7570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517128 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517130 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517133 7570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517136 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517139 7570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517141 7570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517144 7570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517146 7570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517149 7570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517151 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517154 7570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517157 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517160 7570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517162 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517164 7570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:19:48.530113 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517167 7570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517169 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517172 7570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517174 7570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517176 7570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517179 7570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517181 7570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517183 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517187 7570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517191 7570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517193 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517196 7570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517198 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517201 7570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517203 7570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517206 7570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517208 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517211 7570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517213 7570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:19:48.530809 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517216 7570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517218 7570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517220 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517223 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517225 7570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517228 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517230 7570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517232 7570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517235 7570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517238 7570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517240 7570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517242 7570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517245 7570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517247 7570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517250 7570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517252 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517254 7570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517257 7570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517259 7570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517262 7570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:19:48.531367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517264 7570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517270 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517274 7570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517293 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517297 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517301 7570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517304 7570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517306 7570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517309 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517311 7570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517315 7570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517318 7570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517320 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517323 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517327 7570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517330 7570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517333 7570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517336 7570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517339 7570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:19:48.531983 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517342 7570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517344 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517347 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517350 7570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517353 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517355 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517358 7570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517361 7570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517365 7570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517368 7570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517371 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517373 7570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517376 7570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517378 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.517382 7570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.521493 7570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.521510 7570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.521530 7570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.521535 7570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:19:48.532449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521589 7570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521594 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521597 7570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521601 7570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521604 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521607 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521610 7570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521613 7570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521616 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521619 7570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521621 7570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521624 7570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521626 7570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521629 7570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521632 7570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521634 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521637 7570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521640 7570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521642 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521645 7570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:19:48.532989 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521647 7570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521650 7570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521652 7570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521655 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521658 7570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521660 7570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521663 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521667 7570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521669 7570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521674 7570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521676 7570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521679 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521682 7570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521684 7570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521686 7570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521689 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521691 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521694 7570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521696 7570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521698 7570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:19:48.533507 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521701 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521704 7570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521708 7570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521712 7570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521715 7570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521718 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521721 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521724 7570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521726 7570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521729 7570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521731 7570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521734 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521736 7570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521739 7570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521741 7570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521744 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521746 7570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521749 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521751 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:19:48.534008 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521754 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521756 7570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521759 7570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521762 7570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521764 7570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521767 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521769 7570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521773 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521776 7570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521779 7570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521782 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521784 7570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521787 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521790 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521792 7570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521795 7570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521797 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521801 7570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521804 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:19:48.534495 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521808 7570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521810 7570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521813 7570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521815 7570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521818 7570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521820 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521823 7570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521825 7570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521828 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521830 7570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521833 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521835 7570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521838 7570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521840 7570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521843 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521846 7570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521848 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521851 7570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521854 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521856 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:19:48.535025 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521859 7570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521861 7570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521864 7570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521866 7570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521868 7570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521871 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521873 7570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.521876 7570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.521903 7570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522331 7570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522639 7570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522644 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522649 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522653 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522658 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522662 7570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:19:48.535569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522667 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522671 7570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522676 7570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522686 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522691 7570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522695 7570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522699 7570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522703 7570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522708 7570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522714 7570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522719 7570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522724 7570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522732 7570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522739 7570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522747 7570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522757 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522762 7570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522766 7570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522771 7570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:19:48.535973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522776 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522781 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522786 7570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522790 7570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522795 7570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522799 7570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522804 7570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522808 7570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522812 7570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522821 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522827 7570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522833 7570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522837 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522842 7570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522846 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522850 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522854 7570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522858 7570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522862 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:19:48.536441 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522866 7570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522870 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522874 7570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522885 7570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522889 7570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522894 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522898 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522902 7570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522907 7570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522912 7570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522917 7570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522921 7570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522925 7570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522930 7570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522933 7570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522942 7570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522947 7570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522950 7570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522954 7570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522959 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:19:48.537068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522963 7570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522967 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522971 7570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522975 7570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522979 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522983 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522987 7570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.522991 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523000 7570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523005 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523009 7570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523013 7570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523017 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523022 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523026 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523030 7570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523035 7570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523039 7570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523044 7570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:19:48.537587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523048 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523057 7570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523061 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523074 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523078 7570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523082 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523086 7570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523090 7570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523094 7570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523098 7570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523102 7570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523106 7570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523110 7570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523115 7570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523123 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523128 7570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523132 7570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523136 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523140 7570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:19:48.538052 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523144 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523151 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523190 7570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.523359 7570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523409 7570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523415 7570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523420 7570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523424 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523428 7570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523431 7570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523434 7570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523437 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523439 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523442 7570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523445 7570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:19:48.538609 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523447 7570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523450 7570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523453 7570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523477 7570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523482 7570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523485 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523489 7570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523493 7570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523496 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523498 7570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523501 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523504 7570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523506 7570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523509 7570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523512 7570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523514 7570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523517 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523519 7570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523522 7570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523524 7570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:19:48.538978 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523528 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523531 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523534 7570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523536 7570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523539 7570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523542 7570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523544 7570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523547 7570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523550 7570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523552 7570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523555 7570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523558 7570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523560 7570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523563 7570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523567 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523570 7570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523573 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523575 7570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523578 7570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523580 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:19:48.539540 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523582 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523585 7570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523587 7570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523590 7570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523592 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523595 7570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523597 7570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523600 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523603 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523606 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523608 7570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523610 7570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523616 7570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523619 7570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523622 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523624 7570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523627 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523629 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523632 7570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523634 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:19:48.540361 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523637 7570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523639 7570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523642 7570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523644 7570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523648 7570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523651 7570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523653 7570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523656 7570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523658 7570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523661 7570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523663 7570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523666 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523668 7570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523671 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523674 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523677 7570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523679 7570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523682 7570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523685 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:19:48.540986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523687 7570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523690 7570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523692 7570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523695 7570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523697 7570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523701 7570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523704 7570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523706 7570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523709 7570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523711 7570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523714 7570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523716 7570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523719 7570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523723 7570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523725 7570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:19:48.523728 7570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.523781 7570 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.524813 7570 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.524869 7570 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 23:19:48.541449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.525050 7570 server.go:1014] "Starting client certificate rotation" Oct 30 23:19:48.542091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.525179 7570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:19:48.542091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.525231 7570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:13:37 +0000 UTC" deadline="2028-05-08 12:03:53.378493053 +0000 UTC" Oct 30 23:19:48.542091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.525269 7570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13332h44m4.853226119s" Oct 30 23:19:48.542091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.525520 7570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:19:48.542091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.526635 7570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:19:48.542091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.527908 7570 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:19:48.542091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.529848 7570 log.go:25] "Validated CRI v1 image API" Oct 30 23:19:48.542091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.530638 7570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:19:48.542091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.533058 7570 fs.go:135] Filesystem UUIDs: map[0d2073c4-0a4a-4a34-b277-998c64e55246:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a4c4fc0d-419f-4a9e-9c2d-7272f9470270:/dev/nvme0n1p3] Oct 30 23:19:48.542357 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.533073 7570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f36e87ad7c41be1adc49a8e7e4530e9b669abac8e1393642a2ab040ec1d662c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f36e87ad7c41be1adc49a8e7e4530e9b669abac8e1393642a2ab040ec1d662c/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/150395e7507b6aff8a2de8ef1d74767338f9a3bd6850740682852f5395c16d64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/150395e7507b6aff8a2de8ef1d74767338f9a3bd6850740682852f5395c16d64/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a9d86c3ed079b5ced4cf62f22b432c8504dcc3514cfb96275b525e67b595ab6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a9d86c3ed079b5ced4cf62f22b432c8504dcc3514cfb96275b525e67b595ab6/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e17150f06114c7bf69037957484c81ab98d5e99115839018db8c95455d2da51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e17150f06114c7bf69037957484c81ab98d5e99115839018db8c95455d2da51/userdata/shm major:0 minor:428 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fbc8f4e25b798803a0f032a61746747020e245401077974df4d9ee1441460f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fbc8f4e25b798803a0f032a61746747020e245401077974df4d9ee1441460f6/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fdfbfc70ccd4ad2fccc4f11f7983ca667f7426522c07035faec6a82cea360e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fdfbfc70ccd4ad2fccc4f11f7983ca667f7426522c07035faec6a82cea360e5/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/340ebca13d78c6232db1b8859aa77400dd71bc6fbd7b1a6d92de1b53c58c4f02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/340ebca13d78c6232db1b8859aa77400dd71bc6fbd7b1a6d92de1b53c58c4f02/userdata/shm major:0 minor:518 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35b812a727e8b930440d3ccc2bbffc9ce63c1a660e4cd7764db966991701d532/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35b812a727e8b930440d3ccc2bbffc9ce63c1a660e4cd7764db966991701d532/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/478334bb3c33b8c957e40eb40e3d3a7bd2b5cd98ddcf604657dc370754314655/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/478334bb3c33b8c957e40eb40e3d3a7bd2b5cd98ddcf604657dc370754314655/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51f5e9b1a6a87ac27b03fe01101c086817bf1750397b5e8c60565da0b75a4365/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51f5e9b1a6a87ac27b03fe01101c086817bf1750397b5e8c60565da0b75a4365/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52b8ae133b36f50afb97f5e02233ee782fed5ca37c1576a45b50ef098165d960/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52b8ae133b36f50afb97f5e02233ee782fed5ca37c1576a45b50ef098165d960/userdata/shm major:0 minor:312 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52d2af34c7e2a0cad4531f75b651b3cca2d620973340c95b09ae8f0c140747d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52d2af34c7e2a0cad4531f75b651b3cca2d620973340c95b09ae8f0c140747d7/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/691b50e3328cd30fc1ba9c74703894c9120f2fa9e31c063b43e8ee5d3faecdf9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/691b50e3328cd30fc1ba9c74703894c9120f2fa9e31c063b43e8ee5d3faecdf9/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ae127e8f738069b308baef98763318ed6b2cadc9b6e48d415b365e84a5e5214/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ae127e8f738069b308baef98763318ed6b2cadc9b6e48d415b365e84a5e5214/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8573761214d01e14398d2b3c6c7915d7f677e2e65804378f31aa4072d183cf23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8573761214d01e14398d2b3c6c7915d7f677e2e65804378f31aa4072d183cf23/userdata/shm major:0 minor:405 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac67a445adc79e1810ead750a60afb6798db06a1d1a18140f437afeed1c1d6ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac67a445adc79e1810ead750a60afb6798db06a1d1a18140f437afeed1c1d6ec/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac9e25b291a629c4f62445e1df4eb80cdb346647f0f093e8a5ad095dfee7e1d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac9e25b291a629c4f62445e1df4eb80cdb346647f0f093e8a5ad095dfee7e1d7/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b222d723beb7f4f324e9e01fa101c41e1ed6a3d02dc72a4a3c1d68d0ad39054f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b222d723beb7f4f324e9e01fa101c41e1ed6a3d02dc72a4a3c1d68d0ad39054f/userdata/shm major:0 minor:407 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcb715c428511e8227d370f524582b4d843b5afaa559b8d68edcce398c9c28c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcb715c428511e8227d370f524582b4d843b5afaa559b8d68edcce398c9c28c9/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c076674abdd7778f989e75e87b5b1116454b556044f662cd7c5becc8c740bb6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c076674abdd7778f989e75e87b5b1116454b556044f662cd7c5becc8c740bb6a/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6f5c3c59ff333b00077f611922d5d7b2b0d9180a385f4b6d3093dfcf9d53ff2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6f5c3c59ff333b00077f611922d5d7b2b0d9180a385f4b6d3093dfcf9d53ff2/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d96a79c079e48113babe4f1525deef623e4cba5f911392bfe10010af64b455a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d96a79c079e48113babe4f1525deef623e4cba5f911392bfe10010af64b455a7/userdata/shm major:0 minor:349 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0a005ab3d10d39c642af76fa383d89f209ea00f95db3ba5eee5a428839c7409/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0a005ab3d10d39c642af76fa383d89f209ea00f95db3ba5eee5a428839c7409/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec1f1f350e3372f1cfe61c32f39c5694a261ad716e546d94cc74a09fcef92a46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec1f1f350e3372f1cfe61c32f39c5694a261ad716e546d94cc74a09fcef92a46/userdata/shm major:0 minor:512 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecbb743dd6362c36741ee41c3b8ae2d115ec350e24a5567b3b67d3251a24ffd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecbb743dd6362c36741ee41c3b8ae2d115ec350e24a5567b3b67d3251a24ffd3/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efd1d20e544865c8f96659a57ef6fc51bc0898ef703d31aef0bc83efac2b1cc2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efd1d20e544865c8f96659a57ef6fc51bc0898ef703d31aef0bc83efac2b1cc2/userdata/shm major:0 minor:516 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19274393-d63a-4889-8c71-0648e21959a5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/19274393-d63a-4889-8c71-0648e21959a5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19274393-d63a-4889-8c71-0648e21959a5/volumes/kubernetes.io~projected/kube-api-access-9jnn5:{mountpoint:/var/lib/kubelet/pods/19274393-d63a-4889-8c71-0648e21959a5/volumes/kubernetes.io~projected/kube-api-access-9jnn5 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19274393-d63a-4889-8c71-0648e21959a5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/19274393-d63a-4889-8c71-0648e21959a5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f/volumes/kubernetes.io~projected/kube-api-access-f2dsq:{mountpoint:/var/lib/kubelet/pods/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f/volumes/kubernetes.io~projected/kube-api-access-f2dsq major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/284dba08-fe12-4942-a5fd-6c1e2f979a17/volumes/kubernetes.io~projected/kube-api-access-kd7vj:{mountpoint:/var/lib/kubelet/pods/284dba08-fe12-4942-a5fd-6c1e2f979a17/volumes/kubernetes.io~projected/kube-api-access-kd7vj major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/284dba08-fe12-4942-a5fd-6c1e2f979a17/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/284dba08-fe12-4942-a5fd-6c1e2f979a17/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29c3d1e4-09b5-4712-ad6d-9a32434def2d/volumes/kubernetes.io~projected/kube-api-access-n94jk:{mountpoint:/var/lib/kubelet/pods/29c3d1e4-09b5-4712-ad6d-9a32434def2d/volumes/kubernetes.io~projected/kube-api-access-n94jk major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/312d3272-40b0-41c2-8131-bafd6266a11e/volumes/kubernetes.io~projected/kube-api-access-r7ms7:{mountpoint:/var/lib/kubelet/pods/312d3272-40b0-41c2-8131-bafd6266a11e/volumes/kubernetes.io~projected/kube-api-access-r7ms7 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/volumes/kubernetes.io~projected/kube-api-access-qtnkq:{mountpoint:/var/lib/kubelet/pods/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/volumes/kubernetes.io~projected/kube-api-access-qtnkq major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volume-subpaths/web-config/alertmanager/9 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volume-subpaths/web-config/config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~empty-dir/config-out major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~projected/kube-api-access-vwsvr:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~projected/kube-api-access-vwsvr major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~projected/tls-assets major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/config-volume major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/web-config major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68aefc1f-0a32-4722-a494-054f46f44b5b/volumes/kubernetes.io~projected/kube-api-access-mm2q8:{mountpoint:/var/lib/kubelet/pods/68aefc1f-0a32-4722-a494-054f46f44b5b/volumes/kubernetes.io~projected/kube-api-access-mm2q8 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68aefc1f-0a32-4722-a494-054f46f44b5b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/68aefc1f-0a32-4722-a494-054f46f44b5b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68aefc1f-0a32-4722-a494-054f46f44b5b/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/68aefc1f-0a32-4722-a494-054f46f44b5b/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:514 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88de885b-31c5-4ce7-a66f-0a212087affd/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/88de885b-31c5-4ce7-a66f-0a212087affd/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88de885b-31c5-4ce7-a66f-0a212087affd/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/88de885b-31c5-4ce7-a66f-0a212087affd/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88de885b-31c5-4ce7-a66f-0a212087affd/volumes/kubernetes.io~projected/kube-api-access-7v9h2:{mountpoint:/var/lib/kubelet/pods/88de885b-31c5-4ce7-a66f-0a212087affd/volumes/kubernetes.io~projected/kube-api-access-7v9h2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89205e33-cb00-47fe-b78d-cc1a05d1f123/volumes/kubernetes.io~projected/kube-api-access-xhqtj:{mountpoint:/var/lib/kubelet/pods/89205e33-cb00-47fe-b78d-cc1a05d1f123/volumes/kubernetes.io~projected/kube-api-access-xhqtj major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4/volumes/kubernetes.io~projected/kube-api-access-kzvc6:{mountpoint:/var/lib/kubelet/pods/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4/volumes/kubernetes.io~projected/kube-api-access-kzvc6 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4/volumes/kubernetes.io~secret/metrics-certs major:0 minor:508 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97edeeaf-64b2-4628-a669-5fc121c8ee2f/volumes/kubernetes.io~projected/kube-api-access-c6rsj:{mountpoint:/var/lib/kubelet/pods/97edeeaf-64b2-4628-a669-5fc121c8ee2f/volumes/kubernetes.io~projected/kube-api-access-c6rsj major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c20fb11-2dcf-445e-96cd-4d97045a2ab9/volumes/kubernetes.io~projected/kube-api-access-swvkv:{mountpoint:/var/lib/kubelet/pods/9c20fb11-2dcf-445e-96cd-4d97045a2ab9/volumes/kubernetes.io~projected/kube-api-access-swvkv major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c20fb11-2dcf-445e-96cd-4d97045a2ab9/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/9c20fb11-2dcf-445e-96cd-4d97045a2ab9/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0371e4c-ff4b-4867-98a1-bb598ab03845/volumes/kubernetes.io~projected/kube-api-access-mdzpm:{mountpoint:/var/lib/kubelet/pods/a0371e4c-ff4b-4867-98a1-bb598ab03845/volumes/kubernetes.io~projected/kube-api-access-mdzpm major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0371e4c-ff4b-4867-98a1-bb598ab03845/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a0371e4c-ff4b-4867-98a1-bb598ab03845/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0371e4c-ff4b-4867-98a1-bb598ab03845/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a0371e4c-ff4b-4867-98a1-bb598ab03845/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a10c4ed2-0361-47e1-8810-8a710b2be42d/volumes/kubernetes.io~projected/kube-api-access-k8mqf:{mountpoint:/var/lib/kubelet/pods/a10c4ed2-0361-47e1-8810-8a710b2be42d/volumes/kubernetes.io~projected/kube-api-access-k8mqf major:0 minor:515 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5349bf3-d267-4d03-8f45-60e58c5f881e/volumes/kubernetes.io~projected/kube-api-access-9hznz:{mountpoint:/var/lib/kubelet/pods/b5349bf3-d267-4d03-8f45-60e58c5f881e/volumes/kubernetes.io~projected/kube-api-access-9hznz major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5349bf3-d267-4d03-8f45-60e58c5f881e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/b5349bf3-d267-4d03-8f45-60e58c5f881e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5349bf3-d267-4d03-8f45-60e58c5f881e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/b5349bf3-d267-4d03-8f45-60e58c5f881e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb76bb5e-c0a6-4a38-8732-40193dd39cac/volumes/kubernetes.io~projected/kube-api-access-9sjkl:{mountpoint:/var/lib/kubelet/pods/cb76bb5e-c0a6-4a38-8732-40193dd39cac/volumes/kubernetes.io~projected/kube-api-access-9sjkl major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb76bb5e-c0a6-4a38-8732-40193dd39cac/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/cb76bb5e-c0a6-4a38-8732-40193dd39cac/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~projected/kube-api-access-pmvw4:{mountpoint:/var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~projected/kube-api-access-pmvw4 major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~projected/kube-api-access-t5h84:{mountpoint:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~projected/kube-api-access-t5h84 major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d54e0641-5f0d-41bd-8b1b-900463a24cfa/volumes/kubernetes.io~projected/kube-api-access-c76js:{mountpoint:/var/lib/kubelet/pods/d54e0641-5f0d-41bd-8b1b-900463a24cfa/volumes/kubernetes.io~projected/kube-api-access-c76js major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d54e0641-5f0d-41bd-8b1b-900463a24cfa/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/d54e0641-5f0d-41bd-8b1b-900463a24cfa/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volume-subpaths/web-config/config-reloader/2 major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volume-subpaths/web-config/prometheus/15 major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~empty-dir/config-out major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~projected/kube-api-access-zjz76:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~projected/kube-api-access-zjz76 major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~projected/tls-assets major:0 minor:419 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/config major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:418 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:417 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/web-config major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed360b42-1a02-4474-9bf1-2117d1b438f1/volumes/kubernetes.io~projected/kube-api-access-z57gl:{mountpoint:/var/lib/kubelet/pods/ed360b42-1a02-4474-9bf1-2117d1b438f1/volumes/kubernetes.io~projected/kube-api-access-z57gl major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed360b42-1a02-4474-9bf1-2117d1b438f1/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/ed360b42-1a02-4474-9bf1-2117d1b438f1/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed360b42-1a02-4474-9bf1-2117d1b438f1/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/ed360b42-1a02-4474-9bf1-2117d1b438f1/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed360b42-1a02-4474-9bf1-2117d1b438f1/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/ed360b42-1a02-4474-9bf1-2117d1b438f1/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5f12d67-0b46-434f-8c41-a230d421eefd/volumes/kubernetes.io~projected/kube-api-access-lxhqr:{mountpoint:/var/lib/kubelet/pods/f5f12d67-0b46-434f-8c41-a230d421eefd/volumes/kubernetes.io~projected/kube-api-access-lxhqr major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a5c309176eb202bc4611ad5729e9481e5f5dcdc0bc1555e75b609aef9f1793de/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1fd92a325a73f706efe6ce154a5764ec96d70f811c06fd3b10df36b85aaaa079/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b830dad46c70a107f568121442e17d6c57ed98e9fedb48bfcbe803575f44c6b8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/a644fda2555618d5eec515891b49f1c6f690b61a09095fd4f7c8337e3aff41cd/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5dfb9d812d54804edb00ba844f1a5386ca8ec555bcb79c209fbded98159a4438/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b84a05b52289da70061eec39f056a06299c2328aff3d97d572eba8b93ab97836/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4d86043d70c001d12673c73c993554093cf74312485ab5672b56588b1fc45be0/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7cc5070b57b476e4399ea865b3647b685fc83ae06e290144316c4114b9ab2ad3/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/48554265d83ff926268710f46b984028e52680fd6021993e647cc43f95b565f4/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/5faf94b0bcee2a001c75fad5757ab50e1cbe0c0c704256136fb2dec70ee3ea9a/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/7dae0ecfe0d8187b7dbfbb65052d8195244515a6a820bd37827468962332fc52/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/26ac99fb02a37e4926c08c8653a404ff86c95c4d78e127e481ea02248e842af9/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a9bc14739a0a3e7422083cb29e10413933c8853bc30cabdd63e83445b4b6c032/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/5ced9e81568b36ea8a7b027b5a76151c2ec151920cd1f56c22ec859ffd7ca5e4/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0bd751342d0b910bc952cc203f08c3bb411454f4fe1deb6fc808e804fda4ade0/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9a9f4be3a49018715bcb4d1c4d5cb65b74468a2e4e310a505a73f5b8e1792030/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/13fc7ee8866eb5ddedaf4630865a3c9a1891b4ed40a9701fd90e47e992e1bb86/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a49f35122a945905681209db5ea0ab4a8b935a5cb82ae723cc1ff4dd04653201/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/815810e51a6e87afbade1c836ec9b1910d94fe93ec5545786f97c29233ab3971/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/7b8f5dbc0f03541bfbadfe7770e8cf48e4ee208a73aed4b4a3db480d7aa99a1b/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/35271418e1c37de4b7888549ea8717976558ef1cb86ef91daf881eb4bb08bde8/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7f53709d5e0ec310ca86cf207b17f9b29ee0a395e4142fae97c476917d9a5872/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/da822d9d0d2cdc047728088fb162965d14bd57f89e054ac489a8a0951619325f/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/2eeb3bf32e407929b37df7d6fe4af94d541774603eba11d2ccfc64ecab5f460a/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/620ee42d53049268fd47d0f1a39664ca22d1fdff365003b2296756adf18bac33/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/ba41d637e03969830c21052341bcc41c1619da647d62e7beb452718a39bd1e32/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/9a72512fb1e8453ba32c56f101bbfe183acf907cecc90d545d424c7f2798984d/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/39b85c1cd6aa5c71333bd1fa13e6cca6931bdfa3befb1f79f32728b1473e8088/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/f8e3539a2bfe6da4be25bcb30af5428ab1c1d3d6fab81f0b22035429b5c673ee/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/204242ae1cde4dd33bdfafc786b22ed5c635f3c1229b2e586d1533433e1a1de8/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/90de72811eee09b24f55d1f736deab02ddefc2cdb75aec9c5f869a2700cbdf71/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/eb10cfe189a72077416d9595382c191d23f6a1ec9520e0c5ebf60082667efb79/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/c8318d9251d53d1c9742883775a04820c08bfd98162843d62476cf1fb884a8b8/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/be7b50b699869e83a7ac1d65056123c1d082f7c54a2018e3d1b10581ba23001b/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/0dac6c286fb0b5b6447b29ec1f51f2c6c01d9332b32717150983cfb854a43d02/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/c636f2a6b869fe32f6486fbc51f4a95a0f2cee760ad2de3ac00318b2976403ab/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/05fd72b3c91f09e3e27d0f3dcbc88b11565ca12103c50c8ba3883064f99b1120/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/56baec63f55948cea80d82961f0c5560a1f7f4c57db35b0f4a7c40841e4af8b8/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/9f79876b5c8da341e967f5ae2fd88177f82cf63931bbb7a089621385bbd7160b/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/32831e22db5c71996a5a6a89992eeb2442666d85f4470d7dcbaf924fd99550e8/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/a7c0e7c8693fc8a257065b652fd5066c36bccef876a0e04c0ee9f8a197bc99a5/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/485d48f9d9ab722e64040886487d1f3dbdac88493c58ca2ee5902e8e6bb03f3c/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/d56d6140bc7d2e5c56253a9225cca73da7b31f755ae2d850dc9134491ab1586b/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/a404bf6c87b5c53e3878ddb51210498f4557b3e883c2ccd89e7b100ce5a1b5b9/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/84390ea56ec17fe2e4c699696fb2740dfa5f61d67f5ab0c8919a38633f4d4665/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/8fcf25bc01cedf84ba899e5c3eb6261f2177d1fa7eb08356011187d35a0f19a9/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/2b0c9aa16fb6d0257b94d880d9b9304fce30c9a1bc0787f865071ab22fad4c86/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/e402921a279de129be7427fee7b99527e8a0e21029c9e1a5f9fbb74cfcd1e91d/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/cf678d272fc7976b06627168ba82c61f095fb38590fbddc535232b9343eb059f/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/90efbca770046be08c5d002f48b022752d474c92671e020681e23e6db5e7cd98/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/0c09db27888916a85d91c941bc52228061c76fa94b736a8a3dc7e8996ff1a7af/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/c4dcae9a167c26250c827408ebc57ee2fa7496cbfcb1dcf9e64c5f3afad71885/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/8f57ae5b6c03f7a5ffbfd5484509cc17d28ce1246c6ade4d693c20b243a67b6a/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/9be523497e0fe151cfc66edcd237c2e48a0d02540d5fba80d676ec78482569c5/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/0c80fcd28b531c5e3c1f3f58a56620473b8c8d0e5a197cb86cf5f77f2851f98e/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/c64ed1e52846196b1aff36847518c9f757300cf6f4193a110cc752c382125ad7/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/32f4ef24912ac2375218657860eb7c8f91cc7af57d07641a0404d19548193274/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/aead9544bd40cc159d1e6a7d220f9f1cabe7f12e5883de5c718c7299e4a254c0/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/5a5349eee9b0bc6f4fa9ea36707cac98493aa281628d3808950374b0d31b6a9a/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/a255baffe46fcba829c6110007ac4ed6c37edae172e754e6991d3d93804591bb/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/53a2f0ab3acd08f98c4309cd58184f7de269803337be73f113d8fdd5ad4472f1/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/a5aaddb41da4ca8dc2dd02f5f79b7646ab5f42cd545b518de8c38cd2838cba8e/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/30a95659f789fecb08f3df7cf332705eed7af73dfb8015737aada935505bb2be/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/39e54a0677170c09896d6f03393ffd52de086d46bda748378b344b5506f2b596/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/60fd7647cad01e28ecb0b6fa0b7fb4214c64cceae69cc10da083d8561c386c38/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-472:{mountpoint:/var/lib/containers/storage/overlay/873cbd9d8b2d3f4a4daf1a22daaec3ffb0c03312db0d4c0683ebb7aca7696e7a/merged major:0 minor:472 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/beb2779a43a009c6356e35f50542f5321cac999f5dae351ed0adea127279521f/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/a7f81b84613e83a3bf86bdc99730a8ac65af3bb823b46c216c917feb8d203c79/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/ccf9cc9a8f0788e3f609bf98fb0a6f3bd51c7847ffa988af8d88613a99b8d9ba/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/d551dc3cbadeb52c8a5b049bedf918811efef59a638d472aa5ecacd24d78b586/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-498:{mountpoint:/var/lib/containers/storage/overlay/e68cb84172be4a39333cd7abcdfcd538da59c3e7519e017364f0f7c93ea51c65/merged major:0 minor:498 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/75f0108cf238c31c0bcd6dfa87947c79b29edd38a4eeee18c9d56aa443c060c0/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/ed69edde03bd3cf330de915fdf2d1745aaefec3d19f696e00325a04acf427fb9/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-520:{mountpoint:/var/lib/containers/storage/overlay/c796d0120331a58b0facbe8a1539ed8a61b2d6ee4e6aaec7a1b47586a183302e/merged major:0 minor:520 fsType:overlay blockSize:0} overlay_0-522:{mountpoint:/var/lib/containers/storage/overlay/914ba0235fd501e4fba33d6df8e2eed9f63e41a4cb9651121d323ee3ae04c2a4/merged major:0 minor:522 fsType:overlay blockSize:0} overlay_0-524:{mountpoint:/var/lib/containers/storage/overlay/3cb7bf639476a3490301232391482a3517972ba83734b183c99d68019baa3d22/merged major:0 minor:524 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/668e0e3acf19878b8218741e211ca94eaadbea2cc637389ebcfb51a52e223217/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-528:{mountpoint:/var/lib/containers/storage/overlay/f2c679e7cd5f2bfc6b2642f521367d05b8cdda5b25d215a32900f18b69b6ece9/merged major:0 minor:528 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0b16ac04d017400b8610d0c14e3cafc513d682deaa495e2e2936e07419c2f494/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-534:{mountpoint:/var/lib/containers/storage/overlay/47793c876c4c5d01e87c8506c573d65ce5cc0467d2ea9e7fd3d9e89277e28eb8/merged major:0 minor:534 fsType:overlay blockSize:0} overlay_0-539:{mountpoint:/var/lib/containers/storage/overlay/9640cee11f897f54ce412f9f51a65f38b489af493e772d7cc178bb778d3dc805/merged major:0 minor:539 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/268a50d29b6e6e8466bff3ec7625c48e04fb9255017b862b1e14731166cec6f4/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c938542bfe36ebe55a7f032ca35fc8973ffd90385f1d9818467057111b4df8a7/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/48c3ecbd7827f397cb36f71af58e7ca77d8c67fc3b65ec4810205ee6956c68a1/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/669c018af63462489022f768fd3845fd1ea75ec4fa3bdc5917e3927d1fed1d30/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/08611c58bbdf5b145b7040b5ee6ae228b796baee8fb21f34f82f8c7f0f67db58/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/3cd3f4dd4bd95bcc4f16248ff6069a0ec287080492705bd254b02c1f982c4ead/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/760db5d6d5047157f14b5e8449ce74c3b64d5cf1319dc89316361b40f769ce8b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/e71a74d311e5d41ae9c89e958bfc9fea68c3181e34ea558f1e3f3a5f1f9581ef/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 23:19:48.542787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.542300 7570 manager.go:217] Machine: {Timestamp:2026-10-30 23:19:48.541785034 +0000 UTC m=+0.060786116 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3120378 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22e1783cebdd2979b4726d824250c5 SystemUUID:ec22e178-3ceb-dd29-79b4-726d824250c5 BootID:a297dd64-45b4-4e7d-8665-04afd2be3a4b Filesystems:[{Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68aefc1f-0a32-4722-a494-054f46f44b5b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed360b42-1a02-4474-9bf1-2117d1b438f1/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9c20fb11-2dcf-445e-96cd-4d97045a2ab9/volumes/kubernetes.io~projected/kube-api-access-swvkv DeviceMajor:0 DeviceMinor:351 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:418 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:426 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~projected/kube-api-access-zjz76 DeviceMajor:0 DeviceMinor:427 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/691b50e3328cd30fc1ba9c74703894c9120f2fa9e31c063b43e8ee5d3faecdf9/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8573761214d01e14398d2b3c6c7915d7f677e2e65804378f31aa4072d183cf23/userdata/shm DeviceMajor:0 DeviceMinor:405 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a10c4ed2-0361-47e1-8810-8a710b2be42d/volumes/kubernetes.io~projected/kube-api-access-k8mqf DeviceMajor:0 DeviceMinor:515 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/284dba08-fe12-4942-a5fd-6c1e2f979a17/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52b8ae133b36f50afb97f5e02233ee782fed5ca37c1576a45b50ef098165d960/userdata/shm DeviceMajor:0 DeviceMinor:312 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:381 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:419 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68aefc1f-0a32-4722-a494-054f46f44b5b/volumes/kubernetes.io~projected/kube-api-access-mm2q8 DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:285 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:420 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:421 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-472 DeviceMajor:0 DeviceMinor:472 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/478334bb3c33b8c957e40eb40e3d3a7bd2b5cd98ddcf604657dc370754314655/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:306 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ed360b42-1a02-4474-9bf1-2117d1b438f1/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:378 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~projected/kube-api-access-pmvw4 DeviceMajor:0 DeviceMinor:382 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b222d723beb7f4f324e9e01fa101c41e1ed6a3d02dc72a4a3c1d68d0ad39054f/userdata/shm DeviceMajor:0 DeviceMinor:407 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/150395e7507b6aff8a2de8ef1d74767338f9a3bd6850740682852f5395c16d64/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19274393-d63a-4889-8c71-0648e21959a5/volumes/kubernetes.io~projected/kube-api-access-9jnn5 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcb715c428511e8227d370f524582b4d843b5afaa559b8d68edcce398c9c28c9/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:377 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:423 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-524 DeviceMajor:0 DeviceMinor:524 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88de885b-31c5-4ce7-a66f-0a212087affd/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:281 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ed360b42-1a02-4474-9bf1-2117d1b438f1/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:417 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:508 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f5f12d67-0b46-434f-8c41-a230d421eefd/volumes/kubernetes.io~projected/kube-api-access-lxhqr DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~projected/kube-api-access-t5h84 DeviceMajor:0 DeviceMinor:308 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:309 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51f5e9b1a6a87ac27b03fe01101c086817bf1750397b5e8c60565da0b75a4365/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b5349bf3-d267-4d03-8f45-60e58c5f881e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:421 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-534 DeviceMajor:0 DeviceMinor:534 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/284dba08-fe12-4942-a5fd-6c1e2f979a17/volumes/kubernetes.io~projected/kube-api-access-kd7vj DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a0371e4c-ff4b-4867-98a1-bb598ab03845/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68aefc1f-0a32-4722-a494-054f46f44b5b/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a0371e4c-ff4b-4867-98a1-bb598ab03845/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec1f1f350e3372f1cfe61c32f39c5694a261ad716e546d94cc74a09fcef92a46/userdata/shm DeviceMajor:0 DeviceMinor:512 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:416 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/340ebca13d78c6232db1b8859aa77400dd71bc6fbd7b1a6d92de1b53c58c4f02/userdata/shm DeviceMajor:0 DeviceMinor:518 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fdfbfc70ccd4ad2fccc4f11f7983ca667f7426522c07035faec6a82cea360e5/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b5349bf3-d267-4d03-8f45-60e58c5f881e/volumes/kubernetes.io~projected/kube-api-access-9hznz DeviceMajor:0 DeviceMinor:244 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-539 DeviceMajor:0 DeviceMinor:539 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efd1d20e544865c8f96659a57ef6fc51bc0898ef703d31aef0bc83efac2b1cc2/userdata/shm DeviceMajor:0 DeviceMinor:516 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4/volumes/kubernetes.io~projected/kube-api-access-kzvc6 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/89205e33-cb00-47fe-b78d-cc1a05d1f123/volumes/kubernetes.io~projected/kube-api-access-xhqtj DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ae127e8f738069b308baef98763318ed6b2cadc9b6e48d415b365e84a5e5214/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0371e4c-ff4b-4867-98a1-bb598ab03845/volumes/kubernetes.io~projected/kube-api-access-mdzpm DeviceMajor:0 DeviceMinor:235 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fbc8f4e25b798803a0f032a61746747020e245401077974df4d9ee1441460f6/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97edeeaf-64b2-4628-a669-5fc121c8ee2f/volumes/kubernetes.io~projected/kube-api-access-c6rsj DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9c20fb11-2dcf-445e-96cd-4d97045a2ab9/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:399 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:421 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/volumes/kubernetes.io~projected/kube-api-access-qtnkq DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:307 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0b4de16-5f57-44fe-a273-a77ebd48d050/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:376 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6f5c3c59ff333b00077f611922d5d7b2b0d9180a385f4b6d3093dfcf9d53ff2/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:424 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:278 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-522 DeviceMajor:0 DeviceMinor:522 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~projected/kube-api-access-vwsvr DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:514 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:425 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c076674abdd7778f989e75e87b5b1116454b556044f662cd7c5becc8c740bb6a/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:280 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35b812a727e8b930440d3ccc2bbffc9ce63c1a660e4cd7764db966991701d532/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:303 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e17150f06114c7bf69037957484c81ab98d5e99115839018db8c95455d2da51/userdata/shm DeviceMajor:0 DeviceMinor:428 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-498 DeviceMajor:0 DeviceMinor:498 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-528 DeviceMajor:0 DeviceMinor:528 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f36e87ad7c41be1adc49a8e7e4530e9b669abac8e1393642a2ab040ec1d662c/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb76bb5e-c0a6-4a38-8732-40193dd39cac/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:422 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f/volumes/kubernetes.io~projected/kube-api-access-f2dsq DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:304 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52d2af34c7e2a0cad4531f75b651b3cca2d620973340c95b09ae8f0c140747d7/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a9d86c3ed079b5ced4cf62f22b432c8504dcc3514cfb96275b525e67b595ab6/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0a005ab3d10d39c642af76fa383d89f209ea00f95db3ba5eee5a428839c7409/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d189925d-3211-43c6-8bf0-e9aca119e00e/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:305 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ed360b42-1a02-4474-9bf1-2117d1b438f1/volumes/kubernetes.io~projected/kube-api-access-z57gl DeviceMajor:0 DeviceMinor:348 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/88de885b-31c5-4ce7-a66f-0a212087affd/volumes/kubernetes.io~projected/kube-api-access-7v9h2 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d54e0641-5f0d-41bd-8b1b-900463a24cfa/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5349bf3-d267-4d03-8f45-60e58c5f881e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:242 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19274393-d63a-4889-8c71-0648e21959a5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/cb76bb5e-c0a6-4a38-8732-40193dd39cac/volumes/kubernetes.io~projected/kube-api-access-9sjkl DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d96a79c079e48113babe4f1525deef623e4cba5f911392bfe10010af64b455a7/userdata/shm DeviceMajor:0 DeviceMinor:349 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da3a65c4-b342-42d7-aad0-29e63b90125a/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:421 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-520 DeviceMajor:0 DeviceMinor:520 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/29c3d1e4-09b5-4712-ad6d-9a32434def2d/volumes/kubernetes.io~projected/kube-api-access-n94jk DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/312d3272-40b0-41c2-8131-bafd6266a11e/volumes/kubernetes.io~projected/kube-api-access-r7ms7 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d54e0641-5f0d-41bd-8b1b-900463a24cfa/volumes/kubernetes.io~projected/kube-api-access-c76js DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecbb743dd6362c36741ee41c3b8ae2d115ec350e24a5567b3b67d3251a24ffd3/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:258 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19274393-d63a-4889-8c71-0648e21959a5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/88de885b-31c5-4ce7-a66f-0a212087affd/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac67a445adc79e1810ead750a60afb6798db06a1d1a18140f437afeed1c1d6ec/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac9e25b291a629c4f62445e1df4eb80cdb346647f0f093e8a5ad095dfee7e1d7/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5973a913-5fc2-45b7-affa-4184ee2da449/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:150395e7507b6af MacAddress:ce:88:08:6f:b7:2b Speed:10000 Mtu:8901} {Name:1a9d86c3ed079b5 MacAddress:8a:0d:c4:6d:ae:4c Speed:10000 Mtu:8901} {Name:1e17150f06114c7 MacAddress:d6:24:c9:cd:cd:17 Speed:10000 Mtu:8901} {Name:1fbc8f4e25b7988 MacAddress:32:f8:01:27:d9:e8 Speed:10000 Mtu:8901} {Name:340ebca13d78c62 MacAddress:72:78:8f:c2:c8:ea Speed:10000 Mtu:8901} {Name:35b812a727e8b93 MacAddress:b6:4a:fb:70:0b:ab Speed:10000 Mtu:8901} {Name:52b8ae133b36f50 MacAddress:0e:73:82:3f:c0:12 Speed:10000 Mtu:8901} {Name:52d2af34c7e2a0c MacAddress:82:94:66:87:ee:18 Speed:10000 Mtu:8901} {Name:8573761214d01e1 MacAddress:32:c3:35:c8:a2:c0 Speed:10000 Mtu:8901} {Name:ac9e25b291a629c MacAddress:d6:cc:c1:30:8a:20 Speed:10000 Mtu:8901} {Name:b222d723beb7f4f MacAddress:d2:72:4c:af:97:f2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d5:56:c0:89 Speed:0 Mtu:9001} {Name:br-int MacAddress:ce:74:6d:2a:57:40 Speed:0 Mtu:8901} {Name:d96a79c079e4811 MacAddress:02:c1:b9:84:36:79 Speed:10000 Mtu:8901} {Name:e0a005ab3d10d39 MacAddress:be:c6:8b:4e:f5:c6 Speed:10000 Mtu:8901} {Name:ec1f1f350e3372f MacAddress:ce:22:ac:7f:27:ce Speed:10000 Mtu:8901} {Name:efd1d20e544865c MacAddress:7a:e6:39:04:e6:e7 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d5:56:c0:89 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:72:53:b9:96:1f:9d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:27:a2:b9:cd:c8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:19:48.543042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.542793 7570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:19:48.543042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.542840 7570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:19:48.543123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543097 7570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:19:48.543258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543124 7570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-81.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:19:48.543341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543260 7570 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:19:48.543341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543269 7570 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:19:48.543341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543279 7570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:19:48.543341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543300 7570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:19:48.543448 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543384 7570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:19:48.543571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543561 7570 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:19:48.543683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543674 7570 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:19:48.543719 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543711 7570 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:19:48.543747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543723 7570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:19:48.543747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543733 7570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:19:48.543747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543741 7570 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:19:48.543858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.543750 7570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:19:48.544483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.544470 7570 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:19:48.544483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.544483 7570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:19:48.544963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.544947 7570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:19:48.547701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.547688 7570 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:19:48.547701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.547701 7570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:19:48.547806 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.547797 7570 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 23:19:48.547989 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.547979 7570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:19:48.548023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548009 7570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:19:48.548102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548093 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:19:48.548136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548108 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:19:48.548136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548114 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:19:48.548136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548120 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:19:48.548136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548125 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:19:48.548136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548131 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:19:48.548136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548137 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:19:48.548290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548143 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:19:48.548290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548149 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:19:48.548290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548155 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:19:48.548290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548169 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:19:48.548290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548178 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:19:48.548290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548204 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:19:48.548290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.548210 7570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:19:48.551670 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.551655 7570 server.go:1263] "Started kubelet" Oct 30 23:19:48.552437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.551849 7570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:19:48.552437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.552056 7570 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:19:48.552267 ip-10-0-137-81 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:19:48.553042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.552915 7570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:19:48.554486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.554011 7570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:19:48.559512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.558978 7570 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:19:48.559512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.559301 7570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:19:48.559741 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:48.559700 7570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-east-2.compute.internal\" not found" Oct 30 23:19:48.561703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.560239 7570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:19:48.561703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.560260 7570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:19:48.561703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.560250 7570 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:19:48.561703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.560300 7570 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:19:48.561703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.560341 7570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:13:39 +0000 UTC" deadline="2028-06-18 19:56:09.207308843 +0000 UTC" Oct 30 23:19:48.561703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.560388 7570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14324h36m20.646926667s" Oct 30 23:19:48.563097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.561909 7570 factory.go:55] Registering systemd factory Oct 30 23:19:48.563097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.561929 7570 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:19:48.563097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.562638 7570 factory.go:153] Registering CRI-O factory Oct 30 23:19:48.563097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.562651 7570 factory.go:223] Registration of the crio container factory successfully Oct 30 23:19:48.563097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.562716 7570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:19:48.563097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.562736 7570 factory.go:103] Registering Raw factory Oct 30 23:19:48.563097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.562775 7570 manager.go:1199] Started watching for new ooms in manager Oct 30 23:19:48.563439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563111 7570 manager.go:319] Starting recovery of all containers Oct 30 23:19:48.563439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563313 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 23:19:48.563439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563346 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5349bf3-d267-4d03-8f45-60e58c5f881e" volumeName="kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config" seLinuxMountContext="" Oct 30 23:19:48.563439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563362 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-config" seLinuxMountContext="" Oct 30 23:19:48.563439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563378 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed360b42-1a02-4474-9bf1-2117d1b438f1" volumeName="kubernetes.io/configmap/ed360b42-1a02-4474-9bf1-2117d1b438f1-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 23:19:48.563439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563393 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed360b42-1a02-4474-9bf1-2117d1b438f1" volumeName="kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-client-ca-bundle" seLinuxMountContext="" Oct 30 23:19:48.563439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563404 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 23:19:48.563439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563415 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 23:19:48.563439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563427 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97edeeaf-64b2-4628-a669-5fc121c8ee2f" volumeName="kubernetes.io/projected/97edeeaf-64b2-4628-a669-5fc121c8ee2f-kube-api-access-c6rsj" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563443 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b4de16-5f57-44fe-a273-a77ebd48d050" volumeName="kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563474 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d54e0641-5f0d-41bd-8b1b-900463a24cfa" volumeName="kubernetes.io/projected/d54e0641-5f0d-41bd-8b1b-900463a24cfa-kube-api-access-c76js" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563487 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-web-config" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563498 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed360b42-1a02-4474-9bf1-2117d1b438f1" volumeName="kubernetes.io/configmap/ed360b42-1a02-4474-9bf1-2117d1b438f1-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563509 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/configmap/5973a913-5fc2-45b7-affa-4184ee2da449-metrics-client-ca" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563522 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="284dba08-fe12-4942-a5fd-6c1e2f979a17" volumeName="kubernetes.io/projected/284dba08-fe12-4942-a5fd-6c1e2f979a17-kube-api-access-kd7vj" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563536 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/empty-dir/5973a913-5fc2-45b7-affa-4184ee2da449-config-out" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563547 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88de885b-31c5-4ce7-a66f-0a212087affd" volumeName="kubernetes.io/empty-dir/88de885b-31c5-4ce7-a66f-0a212087affd-tmp" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563578 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f" volumeName="kubernetes.io/projected/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-kube-api-access-f2dsq" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563590 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0" volumeName="kubernetes.io/empty-dir/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-volume-directive-shadow" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563601 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/configmap/5973a913-5fc2-45b7-affa-4184ee2da449-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563613 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-web-config" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563627 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68aefc1f-0a32-4722-a494-054f46f44b5b" volumeName="kubernetes.io/secret/68aefc1f-0a32-4722-a494-054f46f44b5b-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563637 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5349bf3-d267-4d03-8f45-60e58c5f881e" volumeName="kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563648 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d189925d-3211-43c6-8bf0-e9aca119e00e" volumeName="kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563668 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563679 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-config-volume" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563691 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563708 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5349bf3-d267-4d03-8f45-60e58c5f881e" volumeName="kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563719 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/projected/da3a65c4-b342-42d7-aad0-29e63b90125a-kube-api-access-zjz76" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563750 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563763 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="312d3272-40b0-41c2-8131-bafd6266a11e" volumeName="kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563775 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0371e4c-ff4b-4867-98a1-bb598ab03845" volumeName="kubernetes.io/secret/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563787 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b4de16-5f57-44fe-a273-a77ebd48d050" volumeName="kubernetes.io/projected/d0b4de16-5f57-44fe-a273-a77ebd48d050-kube-api-access-pmvw4" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563800 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563816 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed360b42-1a02-4474-9bf1-2117d1b438f1" volumeName="kubernetes.io/projected/ed360b42-1a02-4474-9bf1-2117d1b438f1-kube-api-access-z57gl" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563829 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="312d3272-40b0-41c2-8131-bafd6266a11e" volumeName="kubernetes.io/projected/312d3272-40b0-41c2-8131-bafd6266a11e-kube-api-access-r7ms7" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563843 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/empty-dir/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-db" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563856 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19274393-d63a-4889-8c71-0648e21959a5" volumeName="kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-ovnkube-script-lib" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563869 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0" volumeName="kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:19:48.563881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563891 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" volumeName="kubernetes.io/projected/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-kube-api-access-kzvc6" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563904 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb76bb5e-c0a6-4a38-8732-40193dd39cac" volumeName="kubernetes.io/empty-dir/cb76bb5e-c0a6-4a38-8732-40193dd39cac-tmp-dir" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563920 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d54e0641-5f0d-41bd-8b1b-900463a24cfa" volumeName="kubernetes.io/empty-dir/d54e0641-5f0d-41bd-8b1b-900463a24cfa-data-volume" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563933 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0371e4c-ff4b-4867-98a1-bb598ab03845" volumeName="kubernetes.io/empty-dir/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-textfile" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563944 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/empty-dir/5973a913-5fc2-45b7-affa-4184ee2da449-alertmanager-main-db" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563958 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/projected/5973a913-5fc2-45b7-affa-4184ee2da449-tls-assets" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563972 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/projected/5973a913-5fc2-45b7-affa-4184ee2da449-kube-api-access-vwsvr" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563983 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5f12d67-0b46-434f-8c41-a230d421eefd" volumeName="kubernetes.io/configmap/f5f12d67-0b46-434f-8c41-a230d421eefd-serviceca" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.563994 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564007 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c20fb11-2dcf-445e-96cd-4d97045a2ab9" volumeName="kubernetes.io/projected/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-kube-api-access-swvkv" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564019 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0371e4c-ff4b-4867-98a1-bb598ab03845" volumeName="kubernetes.io/configmap/a0371e4c-ff4b-4867-98a1-bb598ab03845-metrics-client-ca" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564031 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b4de16-5f57-44fe-a273-a77ebd48d050" volumeName="kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-metrics-client-ca" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564045 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d189925d-3211-43c6-8bf0-e9aca119e00e" volumeName="kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-grpc-tls" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564057 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564070 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="284dba08-fe12-4942-a5fd-6c1e2f979a17" volumeName="kubernetes.io/secret/284dba08-fe12-4942-a5fd-6c1e2f979a17-cert" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564084 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68aefc1f-0a32-4722-a494-054f46f44b5b" volumeName="kubernetes.io/projected/68aefc1f-0a32-4722-a494-054f46f44b5b-kube-api-access-mm2q8" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564095 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d54e0641-5f0d-41bd-8b1b-900463a24cfa" volumeName="kubernetes.io/secret/d54e0641-5f0d-41bd-8b1b-900463a24cfa-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564106 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19274393-d63a-4889-8c71-0648e21959a5" volumeName="kubernetes.io/projected/19274393-d63a-4889-8c71-0648e21959a5-kube-api-access-9jnn5" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564135 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3eeb220-e1ba-46e2-94c8-2bb19e02dde9" volumeName="kubernetes.io/configmap/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9-konnectivity-ca" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564147 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68aefc1f-0a32-4722-a494-054f46f44b5b" volumeName="kubernetes.io/secret/68aefc1f-0a32-4722-a494-054f46f44b5b-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564159 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5349bf3-d267-4d03-8f45-60e58c5f881e" volumeName="kubernetes.io/empty-dir/b5349bf3-d267-4d03-8f45-60e58c5f881e-tmp" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564173 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d189925d-3211-43c6-8bf0-e9aca119e00e" volumeName="kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564185 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d54e0641-5f0d-41bd-8b1b-900463a24cfa" volumeName="kubernetes.io/configmap/d54e0641-5f0d-41bd-8b1b-900463a24cfa-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564197 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564211 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b4de16-5f57-44fe-a273-a77ebd48d050" volumeName="kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-secret-telemeter-client" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564226 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed360b42-1a02-4474-9bf1-2117d1b438f1" volumeName="kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564239 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29c3d1e4-09b5-4712-ad6d-9a32434def2d" volumeName="kubernetes.io/projected/29c3d1e4-09b5-4712-ad6d-9a32434def2d-kube-api-access-n94jk" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564250 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="312d3272-40b0-41c2-8131-bafd6266a11e" volumeName="kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-cni-binary-copy" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564264 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a10c4ed2-0361-47e1-8810-8a710b2be42d" volumeName="kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564278 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5349bf3-d267-4d03-8f45-60e58c5f881e" volumeName="kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564294 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5349bf3-d267-4d03-8f45-60e58c5f881e" volumeName="kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564305 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0371e4c-ff4b-4867-98a1-bb598ab03845" volumeName="kubernetes.io/projected/a0371e4c-ff4b-4867-98a1-bb598ab03845-kube-api-access-mdzpm" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564316 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19274393-d63a-4889-8c71-0648e21959a5" volumeName="kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-env-overrides" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564330 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cad31bd-cac3-44ca-bb2c-ca302c4e82e4" volumeName="kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564344 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d189925d-3211-43c6-8bf0-e9aca119e00e" volumeName="kubernetes.io/projected/d189925d-3211-43c6-8bf0-e9aca119e00e-kube-api-access-t5h84" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564357 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5f12d67-0b46-434f-8c41-a230d421eefd" volumeName="kubernetes.io/projected/f5f12d67-0b46-434f-8c41-a230d421eefd-kube-api-access-lxhqr" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564370 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b4de16-5f57-44fe-a273-a77ebd48d050" volumeName="kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-federate-client-tls" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564382 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d189925d-3211-43c6-8bf0-e9aca119e00e" volumeName="kubernetes.io/configmap/d189925d-3211-43c6-8bf0-e9aca119e00e-metrics-client-ca" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564393 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564410 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-grpc-tls" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564422 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="312d3272-40b0-41c2-8131-bafd6266a11e" volumeName="kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564436 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f" volumeName="kubernetes.io/empty-dir/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-tmp-dir" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564448 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88de885b-31c5-4ce7-a66f-0a212087affd" volumeName="kubernetes.io/empty-dir/88de885b-31c5-4ce7-a66f-0a212087affd-etc-tuned" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564480 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb76bb5e-c0a6-4a38-8732-40193dd39cac" volumeName="kubernetes.io/projected/cb76bb5e-c0a6-4a38-8732-40193dd39cac-kube-api-access-9sjkl" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564495 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b4de16-5f57-44fe-a273-a77ebd48d050" volumeName="kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564521 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d189925d-3211-43c6-8bf0-e9aca119e00e" volumeName="kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564533 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f" volumeName="kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564546 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89205e33-cb00-47fe-b78d-cc1a05d1f123" volumeName="kubernetes.io/configmap/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-daemon-config" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564559 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564571 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68aefc1f-0a32-4722-a494-054f46f44b5b" volumeName="kubernetes.io/configmap/68aefc1f-0a32-4722-a494-054f46f44b5b-metrics-client-ca" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564583 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88de885b-31c5-4ce7-a66f-0a212087affd" volumeName="kubernetes.io/projected/88de885b-31c5-4ce7-a66f-0a212087affd-kube-api-access-7v9h2" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564597 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b4de16-5f57-44fe-a273-a77ebd48d050" volumeName="kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564611 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d189925d-3211-43c6-8bf0-e9aca119e00e" volumeName="kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564623 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0" volumeName="kubernetes.io/configmap/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-metrics-client-ca" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564635 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97edeeaf-64b2-4628-a669-5fc121c8ee2f" volumeName="kubernetes.io/configmap/97edeeaf-64b2-4628-a669-5fc121c8ee2f-iptables-alerter-script" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564651 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564663 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19274393-d63a-4889-8c71-0648e21959a5" volumeName="kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-ovnkube-config" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564675 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c20fb11-2dcf-445e-96cd-4d97045a2ab9" volumeName="kubernetes.io/secret/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564686 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5349bf3-d267-4d03-8f45-60e58c5f881e" volumeName="kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564701 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb76bb5e-c0a6-4a38-8732-40193dd39cac" volumeName="kubernetes.io/configmap/cb76bb5e-c0a6-4a38-8732-40193dd39cac-config-volume" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564713 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed360b42-1a02-4474-9bf1-2117d1b438f1" volumeName="kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564729 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/empty-dir/da3a65c4-b342-42d7-aad0-29e63b90125a-config-out" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564740 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0371e4c-ff4b-4867-98a1-bb598ab03845" volumeName="kubernetes.io/secret/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-tls" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564753 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0" volumeName="kubernetes.io/projected/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-api-access-qtnkq" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564764 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-cluster-tls-config" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564776 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89205e33-cb00-47fe-b78d-cc1a05d1f123" volumeName="kubernetes.io/configmap/89205e33-cb00-47fe-b78d-cc1a05d1f123-cni-binary-copy" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564791 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d189925d-3211-43c6-8bf0-e9aca119e00e" volumeName="kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564802 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed360b42-1a02-4474-9bf1-2117d1b438f1" volumeName="kubernetes.io/empty-dir/ed360b42-1a02-4474-9bf1-2117d1b438f1-audit-log" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564816 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0" volumeName="kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564831 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b4de16-5f57-44fe-a273-a77ebd48d050" volumeName="kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-telemeter-client-tls" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564844 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564856 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0371e4c-ff4b-4867-98a1-bb598ab03845" volumeName="kubernetes.io/configmap/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564871 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19274393-d63a-4889-8c71-0648e21959a5" volumeName="kubernetes.io/secret/19274393-d63a-4889-8c71-0648e21959a5-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564885 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564914 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d189925d-3211-43c6-8bf0-e9aca119e00e" volumeName="kubernetes.io/empty-dir/d189925d-3211-43c6-8bf0-e9aca119e00e-tmp" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564926 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5973a913-5fc2-45b7-affa-4184ee2da449" volumeName="kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564939 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89205e33-cb00-47fe-b78d-cc1a05d1f123" volumeName="kubernetes.io/projected/89205e33-cb00-47fe-b78d-cc1a05d1f123-kube-api-access-xhqtj" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564957 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5349bf3-d267-4d03-8f45-60e58c5f881e" volumeName="kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564969 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb76bb5e-c0a6-4a38-8732-40193dd39cac" volumeName="kubernetes.io/secret/cb76bb5e-c0a6-4a38-8732-40193dd39cac-metrics-tls" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564980 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3eeb220-e1ba-46e2-94c8-2bb19e02dde9" volumeName="kubernetes.io/secret/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9-agent-certs" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.564995 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.565008 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0" volumeName="kubernetes.io/configmap/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.565019 7570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3a65c4-b342-42d7-aad0-29e63b90125a" volumeName="kubernetes.io/projected/da3a65c4-b342-42d7-aad0-29e63b90125a-tls-assets" seLinuxMountContext="" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.565031 7570 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:19:48.565328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.565038 7570 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:19:48.568875 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.565711 7570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:19:48.568875 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.566953 7570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:19:48.568875 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.568828 7570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:19:48.571685 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.571565 7570 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 23:19:48.592391 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.592322 7570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:19:48.594175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.594151 7570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:19:48.594175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.594175 7570 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:19:48.594348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.594196 7570 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:19:48.594348 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:48.594235 7570 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 23:19:48.598534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.598503 7570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:19:48.605866 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.605837 7570 generic.go:356] "Generic (PLEG): container finished" podID="a0371e4c-ff4b-4867-98a1-bb598ab03845" containerID="fd2d100fea6d36a632a73af0e0a1b200eb7df303efc76914a26439f9b9c1f58d" exitCode=0 Oct 30 23:19:48.618667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.618638 7570 generic.go:356] "Generic (PLEG): container finished" podID="312d3272-40b0-41c2-8131-bafd6266a11e" containerID="16c4d84931e1f719b58d9f373c0f409af8bf57fca3c72b5cc7c217668206c4b1" exitCode=0 Oct 30 23:19:48.618667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.618667 7570 generic.go:356] "Generic (PLEG): container finished" podID="312d3272-40b0-41c2-8131-bafd6266a11e" containerID="badc30416bac8cd0c51aeed0d778396004f52a7b54a351918962dca91433b3f7" exitCode=0 Oct 30 23:19:48.618839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.618677 7570 generic.go:356] "Generic (PLEG): container finished" podID="312d3272-40b0-41c2-8131-bafd6266a11e" containerID="2e1202958373792a5345354eba674fe996b22e7e2dc1757e458a63b4655ef97e" exitCode=0 Oct 30 23:19:48.618839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.618686 7570 generic.go:356] "Generic (PLEG): container finished" podID="312d3272-40b0-41c2-8131-bafd6266a11e" containerID="99e566092093372f977186fb4fd1f21a83de33012e371db21cc72c66b243f209" exitCode=0 Oct 30 23:19:48.618839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.618694 7570 generic.go:356] "Generic (PLEG): container finished" podID="312d3272-40b0-41c2-8131-bafd6266a11e" containerID="b7400a62b0383bf3ac4055fb474864d8b2464bb26cc3762fa47eaa493a3fc9be" exitCode=0 Oct 30 23:19:48.618839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.618701 7570 generic.go:356] "Generic (PLEG): container finished" podID="312d3272-40b0-41c2-8131-bafd6266a11e" containerID="4e3849082a033fe77cfebf63d096e57daec64060097a1f93b0617c4abd84851b" exitCode=0 Oct 30 23:19:48.645180 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.645152 7570 generic.go:356] "Generic (PLEG): container finished" podID="f383e62f25895fcb5217aa3ca64dca1a" containerID="f5c5a3e842ad06feaa5c0c4a9690dd1e8381313d00416c87dceb87359612f142" exitCode=0 Oct 30 23:19:48.649368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.649329 7570 generic.go:356] "Generic (PLEG): container finished" podID="da3a65c4-b342-42d7-aad0-29e63b90125a" containerID="fe5d7d437a54cfb0d94e9119385dc2d4b5ef9244d9e3a1dd605ec3c81565a814" exitCode=0 Oct 30 23:19:48.661844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.661820 7570 generic.go:356] "Generic (PLEG): container finished" podID="5973a913-5fc2-45b7-affa-4184ee2da449" containerID="42e30929878643e9f9096f8cba26a4314c4d0d72f4a3c95105b953f6c19e89b0" exitCode=0 Oct 30 23:19:48.694378 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:48.694308 7570 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 23:19:48.694703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.694687 7570 manager.go:324] Recovery completed Oct 30 23:19:48.722363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722332 7570 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:19:48.722363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722356 7570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:19:48.722573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722382 7570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:19:48.722606 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722571 7570 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 23:19:48.722635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722588 7570 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 23:19:48.722635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722613 7570 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 23:19:48.722635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722622 7570 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 23:19:48.722635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722630 7570 policy_none.go:50] "Start" Oct 30 23:19:48.722743 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722642 7570 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:19:48.722743 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722654 7570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:19:48.722795 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722791 7570 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 23:19:48.722824 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722798 7570 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 23:19:48.722824 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.722804 7570 policy_none.go:44] "Start" Oct 30 23:19:48.727833 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.727801 7570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:19:48.730512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.730500 7570 manager.go:341] "Starting Device Plugin manager" Oct 30 23:19:48.731239 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:48.730525 7570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:19:48.731277 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.731246 7570 server.go:87] "Starting device plugin registration server" Oct 30 23:19:48.731601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.731588 7570 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:19:48.731635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.731603 7570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:19:48.731727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.731705 7570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:19:48.731888 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.731841 7570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:19:48.731888 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.731888 7570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:19:48.732015 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.731894 7570 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:19:48.732015 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.731931 7570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:19:48.731874037Z","UUID":"a58e340b-7260-4c3d-97cd-a707ebe7a46b","Handler":null,"Name":"","Endpoint":""} Oct 30 23:19:48.733428 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:48.733354 7570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:19:48.733819 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.733802 7570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:19:48.733885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.733829 7570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:19:48.832438 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.832397 7570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:19:48.834767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.834747 7570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:19:48.834862 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.834779 7570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:19:48.834862 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.834791 7570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:19:48.834862 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.834841 7570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:48.847412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.847389 7570 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:48.847525 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.847484 7570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:48.895379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.895308 7570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal"] Oct 30 23:19:48.896254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.896239 7570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:48.896434 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.896395 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" event={"ID":"f383e62f25895fcb5217aa3ca64dca1a","Type":"ContainerStarted","Data":"491c22ef68df541bd86e95450c1b6daf6619ff2db7c54f9cee88d18817b81e24"} Oct 30 23:19:48.896498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.896449 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" event={"ID":"f383e62f25895fcb5217aa3ca64dca1a","Type":"ContainerDied","Data":"f5c5a3e842ad06feaa5c0c4a9690dd1e8381313d00416c87dceb87359612f142"} Oct 30 23:19:48.896498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.896489 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" event={"ID":"f383e62f25895fcb5217aa3ca64dca1a","Type":"ContainerStarted","Data":"478334bb3c33b8c957e40eb40e3d3a7bd2b5cd98ddcf604657dc370754314655"} Oct 30 23:19:48.896584 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.896533 7570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:48.896584 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.896553 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" event={"ID":"70da0cbd0028ec04eeaec823ad5f3973","Type":"ContainerStarted","Data":"3d2f7238f76a68abcfd417c3c6034a5cc14904cb8e7cc58d20be8766adaad907"} Oct 30 23:19:48.896584 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.896572 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" event={"ID":"70da0cbd0028ec04eeaec823ad5f3973","Type":"ContainerStarted","Data":"d6f5c3c59ff333b00077f611922d5d7b2b0d9180a385f4b6d3093dfcf9d53ff2"} Oct 30 23:19:48.912364 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.912341 7570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:19:48.912444 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:48.912396 7570 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:48.913071 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.913059 7570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:19:48.913116 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:48.913091 7570 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:48.972863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.972776 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f383e62f25895fcb5217aa3ca64dca1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"f383e62f25895fcb5217aa3ca64dca1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:48.972863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.972812 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f383e62f25895fcb5217aa3ca64dca1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"f383e62f25895fcb5217aa3ca64dca1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:48.972863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:48.972840 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70da0cbd0028ec04eeaec823ad5f3973-config\") pod \"kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"70da0cbd0028ec04eeaec823ad5f3973\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:49.073391 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.073352 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f383e62f25895fcb5217aa3ca64dca1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"f383e62f25895fcb5217aa3ca64dca1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:49.073603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.073395 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f383e62f25895fcb5217aa3ca64dca1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"f383e62f25895fcb5217aa3ca64dca1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:49.073603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.073423 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70da0cbd0028ec04eeaec823ad5f3973-config\") pod \"kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"70da0cbd0028ec04eeaec823ad5f3973\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:49.073603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.073457 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f383e62f25895fcb5217aa3ca64dca1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"f383e62f25895fcb5217aa3ca64dca1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:49.073603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.073523 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f383e62f25895fcb5217aa3ca64dca1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"f383e62f25895fcb5217aa3ca64dca1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:49.073603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.073495 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70da0cbd0028ec04eeaec823ad5f3973-config\") pod \"kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal\" (UID: \"70da0cbd0028ec04eeaec823ad5f3973\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:49.544638 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.544601 7570 apiserver.go:52] "Watching apiserver" Oct 30 23:19:49.554027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.554000 7570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:19:49.554852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.554825 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-nkhth","kube-system/global-pull-secret-syncer-6sfch","openshift-monitoring/metrics-server-5946dc94c7-djbgv","openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb","openshift-image-registry/node-ca-vqvzf","openshift-monitoring/telemeter-client-976bbf6cb-lk9st","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62","openshift-dns/dns-default-h4hmx","openshift-multus/network-metrics-daemon-jgq5s","openshift-ovn-kubernetes/ovnkube-node-562ql","kube-system/konnectivity-agent-89shx","openshift-multus/multus-additional-cni-plugins-lwrmd","openshift-ingress-canary/ingress-canary-9tcb2","openshift-network-operator/iptables-alerter-9nwfw","kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-qmn89","openshift-monitoring/node-exporter-hq9sq","openshift-network-diagnostics/network-check-target-rhm2r","openshift-monitoring/alertmanager-main-0","openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4","openshift-monitoring/monitoring-plugin-77f4f76594-g86k2","openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2","openshift-dns/node-resolver-bwf5h","openshift-insights/insights-runtime-extractor-7vq6t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal","openshift-monitoring/prometheus-k8s-0","openshift-console/console-586c8ddf94-s29tc"] Oct 30 23:19:49.559989 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.559920 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:19:49.560172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.559982 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:19:49.560244 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.560191 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kc52k\"" Oct 30 23:19:49.560605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.560511 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:19:49.561081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.561060 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:19:49.561214 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.561187 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:19:49.561311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.561293 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:19:49.561489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.561335 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:19:49.561550 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.561518 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:19:49.561600 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.561573 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-82xnp\"" Oct 30 23:19:49.561781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.561756 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:19:49.561781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.561764 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lwvlw\"" Oct 30 23:19:49.561943 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.561807 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:19:49.562261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.562245 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:19:49.562603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.562582 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k8dwv\"" Oct 30 23:19:49.562667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.562612 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:19:49.563298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.563277 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:19:49.563588 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.563569 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4bbvk\"" Oct 30 23:19:49.564373 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.564355 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:19:49.564451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.564363 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:19:49.564451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.564442 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2nz9z\"" Oct 30 23:19:49.564679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.564660 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:19:49.564752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.564736 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-djdr2\"" Oct 30 23:19:49.564809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.564773 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:19:49.564809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.564797 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:19:49.565319 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.565171 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:19:49.565319 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.565259 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n9xrj\"" Oct 30 23:19:49.565431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.565341 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:19:49.566588 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.566570 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9tmfn\"" Oct 30 23:19:49.567223 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.567205 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:19:49.567391 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.567378 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:19:49.567482 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.567414 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:19:49.567565 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.567554 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:19:49.567767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.567753 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jt8kr\"" Oct 30 23:19:49.567830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.567814 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:19:49.570343 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.570328 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:19:49.570468 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.570448 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:19:49.570568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.570528 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qmkhv\"" Oct 30 23:19:49.571009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.570975 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:19:49.571143 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571124 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:19:49.571265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571248 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:19:49.571329 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571308 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gskv9\"" Oct 30 23:19:49.571390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571321 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:19:49.571390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571328 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:19:49.571390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571383 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:19:49.571556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571481 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:19:49.571556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571498 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:19:49.571556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571506 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:19:49.571556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571517 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:19:49.571556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571523 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:19:49.571556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571526 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:19:49.571746 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571624 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 23:19:49.571946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.571928 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qwtvf\"" Oct 30 23:19:49.572882 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.572830 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:19:49.573000 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.572930 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:19:49.573227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.573116 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:19:49.573365 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.573308 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:19:49.575832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.575808 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-registration-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.575920 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.575846 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-cni-bin\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.575920 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.575883 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhqtj\" (UniqueName: \"kubernetes.io/projected/89205e33-cb00-47fe-b78d-cc1a05d1f123-kube-api-access-xhqtj\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.576051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.575925 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-tmp-dir\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:19:49.576051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.575959 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.576051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.575990 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-netns\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.576051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576018 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-kubelet\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.576249 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576048 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-multus-certs\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.576249 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576084 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97edeeaf-64b2-4628-a669-5fc121c8ee2f-iptables-alerter-script\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:19:49.576249 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576117 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-cni-multus\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.576249 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576148 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97edeeaf-64b2-4628-a669-5fc121c8ee2f-host-slash\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:19:49.576249 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576177 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2dsq\" (UniqueName: \"kubernetes.io/projected/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-kube-api-access-f2dsq\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:19:49.576249 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576213 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.576249 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576245 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-k8s-cni-cncf-io\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.576602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576276 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-daemon-config\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.576602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576309 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5f12d67-0b46-434f-8c41-a230d421eefd-host\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:19:49.576602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576335 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhqr\" (UniqueName: \"kubernetes.io/projected/f5f12d67-0b46-434f-8c41-a230d421eefd-kube-api-access-lxhqr\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:19:49.576602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576369 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-conf-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.576602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576402 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvc6\" (UniqueName: \"kubernetes.io/projected/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-kube-api-access-kzvc6\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:49.576602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576434 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-socket-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.576602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576492 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94jk\" (UniqueName: \"kubernetes.io/projected/29c3d1e4-09b5-4712-ad6d-9a32434def2d-kube-api-access-n94jk\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.576602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576520 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-system-cni-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.576602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576552 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-cni-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.576602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576586 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89205e33-cb00-47fe-b78d-cc1a05d1f123-cni-binary-copy\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.577070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576618 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-etc-kubernetes\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.577070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576649 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-hosts-file\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:19:49.577070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576678 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mqf\" (UniqueName: \"kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf\") pod \"network-check-target-rhm2r\" (UID: \"a10c4ed2-0361-47e1-8810-8a710b2be42d\") " pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:49.577070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576714 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-device-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.577070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576746 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-hostroot\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.577070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576780 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5f12d67-0b46-434f-8c41-a230d421eefd-serviceca\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:19:49.577070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576812 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-sys-fs\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.577070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576839 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-cnibin\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.577070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576872 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-socket-dir-parent\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.577070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576905 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rsj\" (UniqueName: \"kubernetes.io/projected/97edeeaf-64b2-4628-a669-5fc121c8ee2f-kube-api-access-c6rsj\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:19:49.577070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.576944 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:49.577607 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.577227 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-metrics-certs\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:49.578646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.577713 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-tmp-dir\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:19:49.578809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.578761 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:19:49.579308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579281 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89205e33-cb00-47fe-b78d-cc1a05d1f123-cni-binary-copy\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.579445 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579332 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97edeeaf-64b2-4628-a669-5fc121c8ee2f-iptables-alerter-script\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:19:49.579445 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579391 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 23:19:49.579593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579451 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wczpt\"" Oct 30 23:19:49.579593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579445 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-daemon-config\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.579593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579547 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:19:49.579593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579545 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5f12d67-0b46-434f-8c41-a230d421eefd-serviceca\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:19:49.579769 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579620 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:19:49.579769 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579638 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xj5wh\"" Oct 30 23:19:49.579769 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579709 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:19:49.579928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579876 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hgj94\"" Oct 30 23:19:49.579928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579897 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8bg6s\"" Oct 30 23:19:49.579928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579926 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 23:19:49.580078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579942 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 23:19:49.580078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.579955 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 23:19:49.580078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580007 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 23:19:49.580078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580027 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zcrv2\"" Oct 30 23:19:49.580078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580078 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 23:19:49.580286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580087 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 23:19:49.580286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580086 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 23:19:49.580286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580143 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 23:19:49.580286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580149 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 23:19:49.580286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580175 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 23:19:49.580286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580180 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 23:19:49.580286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580187 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 23:19:49.580286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580200 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 23:19:49.580286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580230 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 23:19:49.580286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580259 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 23:19:49.580683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580665 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z9rpn\"" Oct 30 23:19:49.580821 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580802 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:19:49.580913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580892 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8gxz9\"" Oct 30 23:19:49.580976 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.580940 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 23:19:49.581035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581007 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ww7vc\"" Oct 30 23:19:49.581089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581067 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 23:19:49.581143 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581098 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-anck8g3riltac\"" Oct 30 23:19:49.581143 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581101 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 23:19:49.581143 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581131 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1j1fi1cun0mic\"" Oct 30 23:19:49.581274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581153 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 23:19:49.581274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581244 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 23:19:49.581360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581340 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rpncv\"" Oct 30 23:19:49.581360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581346 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 23:19:49.581453 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581370 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 23:19:49.581453 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581378 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lzqxs\"" Oct 30 23:19:49.581453 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581415 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 23:19:49.581453 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581419 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 23:19:49.581613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581523 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 23:19:49.581613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581591 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 23:19:49.581989 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.581974 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 23:19:49.582030 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.582009 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 23:19:49.582121 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.582107 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xmrvg\"" Oct 30 23:19:49.582182 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.582153 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 23:19:49.582182 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.582168 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 23:19:49.584721 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.584705 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7fh9lroec4t9q\"" Oct 30 23:19:49.584778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.584708 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 23:19:49.587220 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.587203 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhqtj\" (UniqueName: \"kubernetes.io/projected/89205e33-cb00-47fe-b78d-cc1a05d1f123-kube-api-access-xhqtj\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.589476 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.589439 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvc6\" (UniqueName: \"kubernetes.io/projected/8cad31bd-cac3-44ca-bb2c-ca302c4e82e4-kube-api-access-kzvc6\") pod \"network-metrics-daemon-jgq5s\" (UID: \"8cad31bd-cac3-44ca-bb2c-ca302c4e82e4\") " pod="openshift-multus/network-metrics-daemon-jgq5s" Oct 30 23:19:49.589836 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.589822 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8mqf\" (UniqueName: \"kubernetes.io/projected/a10c4ed2-0361-47e1-8810-8a710b2be42d-kube-api-access-k8mqf\") pod \"network-check-target-rhm2r\" (UID: \"a10c4ed2-0361-47e1-8810-8a710b2be42d\") " pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:49.590076 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.590060 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94jk\" (UniqueName: \"kubernetes.io/projected/29c3d1e4-09b5-4712-ad6d-9a32434def2d-kube-api-access-n94jk\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.590975 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.590957 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhqr\" (UniqueName: \"kubernetes.io/projected/f5f12d67-0b46-434f-8c41-a230d421eefd-kube-api-access-lxhqr\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:19:49.591110 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.591093 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2dsq\" (UniqueName: \"kubernetes.io/projected/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-kube-api-access-f2dsq\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:19:49.591161 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.591109 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6rsj\" (UniqueName: \"kubernetes.io/projected/97edeeaf-64b2-4628-a669-5fc121c8ee2f-kube-api-access-c6rsj\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:19:49.659637 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.659606 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 23:19:49.659832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.659652 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 23:19:49.659948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.659676 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 23:19:49.659948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.659696 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 23:19:49.659948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.659710 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 23:19:49.659948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.659749 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 23:19:49.664778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.664756 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 23:19:49.664778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.664776 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 23:19:49.664778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.664787 7570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:19:49.665001 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.664894 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 23:19:49.671103 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.671081 7570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:49.671223 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.671204 7570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:49.677206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677184 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-cni-bin\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.677318 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677214 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-wtmp\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.677318 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677233 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-telemeter-client-tls\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.677318 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677259 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.677318 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677285 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5h84\" (UniqueName: \"kubernetes.io/projected/d189925d-3211-43c6-8bf0-e9aca119e00e-kube-api-access-t5h84\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.677318 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677289 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-cni-bin\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.677524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677344 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5973a913-5fc2-45b7-affa-4184ee2da449-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.677524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677420 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.677524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677447 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-netns\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.677524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677504 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-netns\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.677524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677507 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7ms7\" (UniqueName: \"kubernetes.io/projected/312d3272-40b0-41c2-8131-bafd6266a11e-kube-api-access-r7ms7\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.677780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677540 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-sys\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.677780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677544 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.677780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677566 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb76bb5e-c0a6-4a38-8732-40193dd39cac-tmp-dir\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:49.677780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677590 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da3a65c4-b342-42d7-aad0-29e63b90125a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.677780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677614 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.677780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677654 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.677780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677670 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5973a913-5fc2-45b7-affa-4184ee2da449-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.677780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677703 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-telemeter-client-tls\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.677780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677701 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.677780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677742 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97edeeaf-64b2-4628-a669-5fc121c8ee2f-host-slash\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677815 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb76bb5e-c0a6-4a38-8732-40193dd39cac-tmp-dir\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677830 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677872 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677880 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97edeeaf-64b2-4628-a669-5fc121c8ee2f-host-slash\") pod \"iptables-alerter-9nwfw\" (UID: \"97edeeaf-64b2-4628-a669-5fc121c8ee2f\") " pod="openshift-network-operator/iptables-alerter-9nwfw" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677902 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-accelerators-collector-config\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677931 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-systemd\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677954 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88de885b-31c5-4ce7-a66f-0a212087affd-tmp\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677961 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da3a65c4-b342-42d7-aad0-29e63b90125a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.677980 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-systemd\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678006 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678038 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-cni-netd\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678059 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678063 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88de885b-31c5-4ce7-a66f-0a212087affd-tmp\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678073 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-ovnkube-script-lib\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678102 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-tls\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678119 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-accelerators-collector-config\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678175 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678234 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-k8s-cni-cncf-io\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.678261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678253 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678285 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d54e0641-5f0d-41bd-8b1b-900463a24cfa-data-volume\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678295 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-ovnkube-script-lib\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678313 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-k8s-cni-cncf-io\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678336 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed360b42-1a02-4474-9bf1-2117d1b438f1-audit-log\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678362 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678366 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-tls\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678372 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d54e0641-5f0d-41bd-8b1b-900463a24cfa-data-volume\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678395 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678410 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed360b42-1a02-4474-9bf1-2117d1b438f1-audit-log\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678420 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-secret-metrics-server-tls\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678477 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-kubelet-config\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678548 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678579 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb76bb5e-c0a6-4a38-8732-40193dd39cac-metrics-tls\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678592 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678607 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678632 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5973a913-5fc2-45b7-affa-4184ee2da449-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678654 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-socket-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678660 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-secret-metrics-server-tls\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678676 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-system-cni-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678700 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-etc-kubernetes\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678709 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5973a913-5fc2-45b7-affa-4184ee2da449-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678730 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-serving-certs-ca-bundle\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678754 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-system-cni-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678756 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88de885b-31c5-4ce7-a66f-0a212087affd-etc-tuned\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678764 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb76bb5e-c0a6-4a38-8732-40193dd39cac-metrics-tls\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678793 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-socket-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678802 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb76bb5e-c0a6-4a38-8732-40193dd39cac-config-volume\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678818 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-etc-kubernetes\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678827 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/284dba08-fe12-4942-a5fd-6c1e2f979a17-cert\") pod \"ingress-canary-9tcb2\" (UID: \"284dba08-fe12-4942-a5fd-6c1e2f979a17\") " pod="openshift-ingress-canary/ingress-canary-9tcb2" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678852 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-g86k2\" (UID: \"9c20fb11-2dcf-445e-96cd-4d97045a2ab9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678862 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678875 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-device-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678884 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88de885b-31c5-4ce7-a66f-0a212087affd-etc-tuned\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678901 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-hostroot\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678905 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-serving-certs-ca-bundle\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678925 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-dbus\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678943 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-device-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678950 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.678979 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679008 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd7vj\" (UniqueName: \"kubernetes.io/projected/284dba08-fe12-4942-a5fd-6c1e2f979a17-kube-api-access-kd7vj\") pod \"ingress-canary-9tcb2\" (UID: \"284dba08-fe12-4942-a5fd-6c1e2f979a17\") " pod="openshift-ingress-canary/ingress-canary-9tcb2" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679036 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-kubernetes\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679054 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb76bb5e-c0a6-4a38-8732-40193dd39cac-config-volume\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679058 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-lib-modules\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679063 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/284dba08-fe12-4942-a5fd-6c1e2f979a17-cert\") pod \"ingress-canary-9tcb2\" (UID: \"284dba08-fe12-4942-a5fd-6c1e2f979a17\") " pod="openshift-ingress-canary/ingress-canary-9tcb2" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679076 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-g86k2\" (UID: \"9c20fb11-2dcf-445e-96cd-4d97045a2ab9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679082 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.679152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679104 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5349bf3-d267-4d03-8f45-60e58c5f881e-tmp\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679191 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5349bf3-d267-4d03-8f45-60e58c5f881e-tmp\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679212 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679253 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679277 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-hostroot\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679303 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679314 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvkv\" (UniqueName: \"kubernetes.io/projected/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-kube-api-access-swvkv\") pod \"monitoring-plugin-77f4f76594-g86k2\" (UID: \"9c20fb11-2dcf-445e-96cd-4d97045a2ab9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679332 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679344 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.679357 7570 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.679494 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:50.179450724 +0000 UTC m=+1.698451808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679603 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679620 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-federate-client-tls\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679645 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679662 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679687 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679711 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-ovnkube-config\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679734 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679743 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679758 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5973a913-5fc2-45b7-affa-4184ee2da449-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679788 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwsvr\" (UniqueName: \"kubernetes.io/projected/5973a913-5fc2-45b7-affa-4184ee2da449-kube-api-access-vwsvr\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.679811 7570 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679815 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-run\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679838 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-federate-client-tls\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679858 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-slash\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679886 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.679897 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:50.179885162 +0000 UTC m=+1.698886265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679912 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5973a913-5fc2-45b7-affa-4184ee2da449-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679911 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679919 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5f12d67-0b46-434f-8c41-a230d421eefd-host\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679936 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-ovnkube-config\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679951 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5f12d67-0b46-434f-8c41-a230d421eefd-host\") pod \"node-ca-vqvzf\" (UID: \"f5f12d67-0b46-434f-8c41-a230d421eefd\") " pod="openshift-image-registry/node-ca-vqvzf" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679951 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.679994 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-metrics-client-ca\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680027 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680057 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680063 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680083 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680107 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-root\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680134 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-metrics-client-ca\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680132 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-secret-metrics-server-client-certs\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680179 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da3a65c4-b342-42d7-aad0-29e63b90125a-config-out\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680209 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680232 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9-agent-certs\") pod \"konnectivity-agent-89shx\" (UID: \"d3eeb220-e1ba-46e2-94c8-2bb19e02dde9\") " pod="kube-system/konnectivity-agent-89shx" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680238 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da3a65c4-b342-42d7-aad0-29e63b90125a-config-out\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680249 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680271 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-cni-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680280 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680291 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-hosts-file\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680311 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed360b42-1a02-4474-9bf1-2117d1b438f1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680328 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-secret-metrics-server-client-certs\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680336 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d54e0641-5f0d-41bd-8b1b-900463a24cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680349 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f-hosts-file\") pod \"node-resolver-bwf5h\" (UID: \"1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f\") " pod="openshift-dns/node-resolver-bwf5h" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680353 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680371 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680407 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-telemeter-trusted-ca-bundle\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680429 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680438 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvw4\" (UniqueName: \"kubernetes.io/projected/d0b4de16-5f57-44fe-a273-a77ebd48d050-kube-api-access-pmvw4\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680439 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-cni-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680514 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed360b42-1a02-4474-9bf1-2117d1b438f1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680526 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysconfig\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680558 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680560 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680586 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9-agent-certs\") pod \"konnectivity-agent-89shx\" (UID: \"d3eeb220-e1ba-46e2-94c8-2bb19e02dde9\") " pod="kube-system/konnectivity-agent-89shx" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680623 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-web-config\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680674 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-sys-fs\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680688 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680693 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-cnibin\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680717 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d54e0641-5f0d-41bd-8b1b-900463a24cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680727 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-cnibin\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680730 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtnkq\" (UniqueName: \"kubernetes.io/projected/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-api-access-qtnkq\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680730 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-sys-fs\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680753 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-var-lib-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680794 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680795 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-web-config\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680819 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5973a913-5fc2-45b7-affa-4184ee2da449-config-out\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680835 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-var-lib-kubelet\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680851 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-multus-certs\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680865 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-host\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680879 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.680895 7570 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680901 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5973a913-5fc2-45b7-affa-4184ee2da449-config-out\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680915 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680935 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-cni-multus\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680943 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-run-multus-certs\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680945 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0b4de16-5f57-44fe-a273-a77ebd48d050-telemeter-trusted-ca-bundle\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680953 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-client-ca-bundle\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.680972 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:50.180956342 +0000 UTC m=+1.699957436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-serving-cert" not registered Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.680983 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-cni-multus\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681046 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sjkl\" (UniqueName: \"kubernetes.io/projected/cb76bb5e-c0a6-4a38-8732-40193dd39cac-kube-api-access-9sjkl\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681069 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681099 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d54e0641-5f0d-41bd-8b1b-900463a24cfa-crio-socket\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681124 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-secret-telemeter-client\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681148 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-modprobe-d\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681170 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681175 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681196 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed360b42-1a02-4474-9bf1-2117d1b438f1-client-ca-bundle\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681200 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-conf-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681249 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-original-pull-secret\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681248 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-conf-dir\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681263 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681286 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-cnibin\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681333 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d0b4de16-5f57-44fe-a273-a77ebd48d050-secret-telemeter-client\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681331 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysctl-conf\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681369 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9-konnectivity-ca\") pod \"konnectivity-agent-89shx\" (UID: \"d3eeb220-e1ba-46e2-94c8-2bb19e02dde9\") " pod="kube-system/konnectivity-agent-89shx" Oct 30 23:19:49.681324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681385 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681402 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681419 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/68aefc1f-0a32-4722-a494-054f46f44b5b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681434 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjz76\" (UniqueName: \"kubernetes.io/projected/da3a65c4-b342-42d7-aad0-29e63b90125a-kube-api-access-zjz76\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681454 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-etc-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681546 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-e1ba-46e2-94c8-2bb19e02dde9-konnectivity-ca\") pod \"konnectivity-agent-89shx\" (UID: \"d3eeb220-e1ba-46e2-94c8-2bb19e02dde9\") " pod="kube-system/konnectivity-agent-89shx" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681569 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-node-log\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681593 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681609 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681629 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d54e0641-5f0d-41bd-8b1b-900463a24cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681643 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/312d3272-40b0-41c2-8131-bafd6266a11e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681651 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/68aefc1f-0a32-4722-a494-054f46f44b5b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.681662 7570 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681656 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68aefc1f-0a32-4722-a494-054f46f44b5b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.681682 7570 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681693 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.681711 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:50.181696063 +0000 UTC m=+1.700697139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-config" not registered Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.681731 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:50.181719585 +0000 UTC m=+1.700720653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"service-ca" not registered Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681744 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-socket-dir-parent\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681770 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed360b42-1a02-4474-9bf1-2117d1b438f1-metrics-server-audit-profiles\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681788 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysctl-d\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681804 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-grpc-tls\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681806 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-multus-socket-dir-parent\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681788 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68aefc1f-0a32-4722-a494-054f46f44b5b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681826 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-registration-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681860 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d54e0641-5f0d-41bd-8b1b-900463a24cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681868 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-system-cni-dir\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681886 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29c3d1e4-09b5-4712-ad6d-9a32434def2d-registration-dir\") pod \"aws-ebs-csi-driver-node-xsv62\" (UID: \"29c3d1e4-09b5-4712-ad6d-9a32434def2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681906 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681933 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-run-netns\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681947 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed360b42-1a02-4474-9bf1-2117d1b438f1-metrics-server-audit-profiles\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681961 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hznz\" (UniqueName: \"kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681974 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d189925d-3211-43c6-8bf0-e9aca119e00e-secret-grpc-tls\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.681983 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-kubelet\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682006 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89205e33-cb00-47fe-b78d-cc1a05d1f123-host-var-lib-kubelet\") pod \"multus-nkhth\" (UID: \"89205e33-cb00-47fe-b78d-cc1a05d1f123\") " pod="openshift-multus/multus-nkhth" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682007 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682027 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682044 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-cni-bin\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682071 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdzpm\" (UniqueName: \"kubernetes.io/projected/a0371e4c-ff4b-4867-98a1-bb598ab03845-kube-api-access-mdzpm\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682098 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682136 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682172 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-kubelet\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682180 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682205 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682245 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0371e4c-ff4b-4867-98a1-bb598ab03845-metrics-client-ca\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682257 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682274 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z57gl\" (UniqueName: \"kubernetes.io/projected/ed360b42-1a02-4474-9bf1-2117d1b438f1-kube-api-access-z57gl\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682301 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682328 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-systemd-units\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682351 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-env-overrides\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682390 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-textfile\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682419 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d189925d-3211-43c6-8bf0-e9aca119e00e-tmp\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682447 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682451 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0371e4c-ff4b-4867-98a1-bb598ab03845-metrics-client-ca\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682447 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682509 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-textfile\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682517 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-sys\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682519 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19274393-d63a-4889-8c71-0648e21959a5-env-overrides\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682536 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d189925d-3211-43c6-8bf0-e9aca119e00e-tmp\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682558 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682559 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9h2\" (UniqueName: \"kubernetes.io/projected/88de885b-31c5-4ce7-a66f-0a212087affd-kube-api-access-7v9h2\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682593 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68aefc1f-0a32-4722-a494-054f46f44b5b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682617 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d189925d-3211-43c6-8bf0-e9aca119e00e-metrics-client-ca\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682715 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682739 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682768 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-ovn\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682785 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d189925d-3211-43c6-8bf0-e9aca119e00e-metrics-client-ca\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682794 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-log-socket\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682803 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68aefc1f-0a32-4722-a494-054f46f44b5b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682833 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682855 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-tls\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682870 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-config\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682886 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jnn5\" (UniqueName: \"kubernetes.io/projected/19274393-d63a-4889-8c71-0648e21959a5-kube-api-access-9jnn5\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682910 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682937 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.682959 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-web-config\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683038 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5973a913-5fc2-45b7-affa-4184ee2da449-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683057 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683068 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76js\" (UniqueName: \"kubernetes.io/projected/d54e0641-5f0d-41bd-8b1b-900463a24cfa-kube-api-access-c76js\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683094 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm2q8\" (UniqueName: \"kubernetes.io/projected/68aefc1f-0a32-4722-a494-054f46f44b5b-kube-api-access-mm2q8\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683113 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-tls\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683123 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683164 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19274393-d63a-4889-8c71-0648e21959a5-ovn-node-metrics-cert\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683221 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-config-volume\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683271 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683339 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-web-config\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683389 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/da3a65c4-b342-42d7-aad0-29e63b90125a-config\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683494 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5973a913-5fc2-45b7-affa-4184ee2da449-config-volume\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.683574 7570 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683608 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.683625 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:50.183610809 +0000 UTC m=+1.702611899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-oauth-config" not registered Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683676 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5973a913-5fc2-45b7-affa-4184ee2da449-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683717 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19274393-d63a-4889-8c71-0648e21959a5-ovn-node-metrics-cert\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.683758 7570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:19:49.685454 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.683835 7570 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:49.688016 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.686087 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da3a65c4-b342-42d7-aad0-29e63b90125a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.695548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.695527 7570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:19:49.695633 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.695577 7570 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal" Oct 30 23:19:49.696200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.696123 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sjkl\" (UniqueName: \"kubernetes.io/projected/cb76bb5e-c0a6-4a38-8732-40193dd39cac-kube-api-access-9sjkl\") pod \"dns-default-h4hmx\" (UID: \"cb76bb5e-c0a6-4a38-8732-40193dd39cac\") " pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:49.696322 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.696274 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvkv\" (UniqueName: \"kubernetes.io/projected/9c20fb11-2dcf-445e-96cd-4d97045a2ab9-kube-api-access-swvkv\") pod \"monitoring-plugin-77f4f76594-g86k2\" (UID: \"9c20fb11-2dcf-445e-96cd-4d97045a2ab9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:49.696404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.696374 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm2q8\" (UniqueName: \"kubernetes.io/projected/68aefc1f-0a32-4722-a494-054f46f44b5b-kube-api-access-mm2q8\") pod \"openshift-state-metrics-7bd7489957-t49gb\" (UID: \"68aefc1f-0a32-4722-a494-054f46f44b5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" Oct 30 23:19:49.696404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.696379 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7ms7\" (UniqueName: \"kubernetes.io/projected/312d3272-40b0-41c2-8131-bafd6266a11e-kube-api-access-r7ms7\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.696595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.696535 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjz76\" (UniqueName: \"kubernetes.io/projected/da3a65c4-b342-42d7-aad0-29e63b90125a-kube-api-access-zjz76\") pod \"prometheus-k8s-0\" (UID: \"da3a65c4-b342-42d7-aad0-29e63b90125a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:49.696595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.696568 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtnkq\" (UniqueName: \"kubernetes.io/projected/3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0-kube-api-access-qtnkq\") pod \"kube-state-metrics-5dc8d7d89b-thls4\" (UID: \"3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" Oct 30 23:19:49.696595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.696576 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwsvr\" (UniqueName: \"kubernetes.io/projected/5973a913-5fc2-45b7-affa-4184ee2da449-kube-api-access-vwsvr\") pod \"alertmanager-main-0\" (UID: \"5973a913-5fc2-45b7-affa-4184ee2da449\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:19:49.696808 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.696693 7570 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 23:19:49.696808 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.696712 7570 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 23:19:49.696808 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.696742 7570 projected.go:196] Error preparing data for projected volume kube-api-access-9hznz for pod openshift-console/console-586c8ddf94-s29tc: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 23:19:49.696958 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:49.696819 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:50.196788042 +0000 UTC m=+1.715789128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9hznz" (UniqueName: "kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 23:19:49.696958 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.696867 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jnn5\" (UniqueName: \"kubernetes.io/projected/19274393-d63a-4889-8c71-0648e21959a5-kube-api-access-9jnn5\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.696958 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.696892 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9h2\" (UniqueName: \"kubernetes.io/projected/88de885b-31c5-4ce7-a66f-0a212087affd-kube-api-access-7v9h2\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.696958 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.696869 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76js\" (UniqueName: \"kubernetes.io/projected/d54e0641-5f0d-41bd-8b1b-900463a24cfa-kube-api-access-c76js\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:49.697177 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.697031 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z57gl\" (UniqueName: \"kubernetes.io/projected/ed360b42-1a02-4474-9bf1-2117d1b438f1-kube-api-access-z57gl\") pod \"metrics-server-5946dc94c7-djbgv\" (UID: \"ed360b42-1a02-4474-9bf1-2117d1b438f1\") " pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:49.697334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.697316 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5h84\" (UniqueName: \"kubernetes.io/projected/d189925d-3211-43c6-8bf0-e9aca119e00e-kube-api-access-t5h84\") pod \"thanos-querier-5f65cc44c7-ljdt2\" (UID: \"d189925d-3211-43c6-8bf0-e9aca119e00e\") " pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:49.697418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.697358 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvw4\" (UniqueName: \"kubernetes.io/projected/d0b4de16-5f57-44fe-a273-a77ebd48d050-kube-api-access-pmvw4\") pod \"telemeter-client-976bbf6cb-lk9st\" (UID: \"d0b4de16-5f57-44fe-a273-a77ebd48d050\") " pod="openshift-monitoring/telemeter-client-976bbf6cb-lk9st" Oct 30 23:19:49.697451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.697420 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd7vj\" (UniqueName: \"kubernetes.io/projected/284dba08-fe12-4942-a5fd-6c1e2f979a17-kube-api-access-kd7vj\") pod \"ingress-canary-9tcb2\" (UID: \"284dba08-fe12-4942-a5fd-6c1e2f979a17\") " pod="openshift-ingress-canary/ingress-canary-9tcb2" Oct 30 23:19:49.698336 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.698318 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdzpm\" (UniqueName: \"kubernetes.io/projected/a0371e4c-ff4b-4867-98a1-bb598ab03845-kube-api-access-mdzpm\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.783677 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783643 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysconfig\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.783677 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783682 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-var-lib-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.783909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783716 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-var-lib-kubelet\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.783909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783738 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-host\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.783909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783763 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d54e0641-5f0d-41bd-8b1b-900463a24cfa-crio-socket\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:49.783909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783785 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-modprobe-d\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.783909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783780 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysconfig\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.783909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783824 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-host\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.783909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783831 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-var-lib-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.783909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783834 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-var-lib-kubelet\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.783909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783856 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d54e0641-5f0d-41bd-8b1b-900463a24cfa-crio-socket\") pod \"insights-runtime-extractor-7vq6t\" (UID: \"d54e0641-5f0d-41bd-8b1b-900463a24cfa\") " pod="openshift-insights/insights-runtime-extractor-7vq6t" Oct 30 23:19:49.783909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783854 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.783909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783893 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783911 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-modprobe-d\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783951 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-cnibin\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.783981 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysctl-conf\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784005 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-etc-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784023 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-node-log\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784051 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-node-log\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784026 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-cnibin\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784056 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-etc-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784078 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysctl-d\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784105 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-system-cni-dir\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784125 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-run-netns\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784142 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysctl-conf\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784157 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-system-cni-dir\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784161 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-cni-bin\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784191 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-kubelet\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784202 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-cni-bin\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784192 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-run-netns\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784221 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-systemd-units\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784226 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-sysctl-d\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784223 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-kubelet\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784244 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-sys\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784273 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-sys\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784283 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-ovn\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784302 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-systemd-units\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784317 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-ovn\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784328 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-log-socket\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784307 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-log-socket\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.784355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784361 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784410 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-wtmp\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784441 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-sys\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784488 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-node-exporter-wtmp\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784489 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-systemd\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784453 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784518 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-systemd\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784531 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-sys\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784541 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-cni-netd\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784548 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-systemd\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784570 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-cni-netd\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784579 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-systemd\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784680 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-kubelet-config\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.784579 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-kubelet-config\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785031 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-dbus\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785058 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-kubernetes\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785080 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-lib-modules\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785109 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-etc-kubernetes\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785124 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785159 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-run-openvswitch\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785162 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-run\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785124 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f-dbus\") pod \"global-pull-secret-syncer-6sfch\" (UID: \"6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f\") " pod="kube-system/global-pull-secret-syncer-6sfch" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785195 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-slash\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785199 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-lib-modules\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785207 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88de885b-31c5-4ce7-a66f-0a212087affd-run\") pod \"tuned-qmn89\" (UID: \"88de885b-31c5-4ce7-a66f-0a212087affd\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn89" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785228 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19274393-d63a-4889-8c71-0648e21959a5-host-slash\") pod \"ovnkube-node-562ql\" (UID: \"19274393-d63a-4889-8c71-0648e21959a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785231 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785259 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-root\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785311 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0371e4c-ff4b-4867-98a1-bb598ab03845-root\") pod \"node-exporter-hq9sq\" (UID: \"a0371e4c-ff4b-4867-98a1-bb598ab03845\") " pod="openshift-monitoring/node-exporter-hq9sq" Oct 30 23:19:49.785497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:49.785325 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/312d3272-40b0-41c2-8131-bafd6266a11e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwrmd\" (UID: \"312d3272-40b0-41c2-8131-bafd6266a11e\") " pod="openshift-multus/multus-additional-cni-plugins-lwrmd" Oct 30 23:19:50.189570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:50.189534 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:50.189570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:50.189569 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:50.189790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:50.189613 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:50.189790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:50.189658 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:50.189790 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.189658 7570 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 23:19:50.189790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:50.189683 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:50.189790 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.189720 7570 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 23:19:50.189790 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.189735 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:51.189714158 +0000 UTC m=+2.708715247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-config" not registered Oct 30 23:19:50.189790 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.189752 7570 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 23:19:50.189790 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.189759 7570 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 23:19:50.189790 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.189775 7570 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 23:19:50.189790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:50.189777 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:50.189790 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.189782 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:51.189770789 +0000 UTC m=+2.708771858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"service-ca" not registered Oct 30 23:19:50.190097 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.189811 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:51.189800279 +0000 UTC m=+2.708801351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 23:19:50.190097 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.189827 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:51.189820874 +0000 UTC m=+2.708821942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 23:19:50.190097 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.189839 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:51.189833682 +0000 UTC m=+2.708834750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-oauth-config" not registered Oct 30 23:19:50.190097 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.189857 7570 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 23:19:50.190097 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.189906 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:51.189892587 +0000 UTC m=+2.708893683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-serving-cert" not registered Oct 30 23:19:50.291144 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:50.291107 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hznz\" (UniqueName: \"kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:50.291311 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.291290 7570 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 23:19:50.291349 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.291325 7570 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 23:19:50.291349 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.291336 7570 projected.go:196] Error preparing data for projected volume kube-api-access-9hznz for pod openshift-console/console-586c8ddf94-s29tc: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 23:19:50.291431 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:50.291387 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:51.291373533 +0000 UTC m=+2.810374626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hznz" (UniqueName: "kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 23:19:50.434237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:50.434200 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:19:51.200384 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:51.200352 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:51.200384 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:51.200386 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:51.200419 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:51.200445 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:51.200488 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.200502 7570 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:51.200548 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.200567 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:53.200553668 +0000 UTC m=+4.719554799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.200579 7570 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.200502 7570 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.200592 7570 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.200617 7570 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.200622 7570 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.200638 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:53.200623299 +0000 UTC m=+4.719624370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-serving-cert" not registered Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.200670 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:53.200647514 +0000 UTC m=+4.719648596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.200703 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:53.200688731 +0000 UTC m=+4.719689798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-config" not registered Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.200725 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:53.200716293 +0000 UTC m=+4.719717361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"service-ca" not registered Oct 30 23:19:51.200905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.200745 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:53.200737298 +0000 UTC m=+4.719738366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-oauth-config" not registered Oct 30 23:19:51.301821 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:51.301783 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hznz\" (UniqueName: \"kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:51.301981 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.301947 7570 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 23:19:51.301981 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.301970 7570 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 23:19:51.301981 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.301980 7570 projected.go:196] Error preparing data for projected volume kube-api-access-9hznz for pod openshift-console/console-586c8ddf94-s29tc: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 23:19:51.302111 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:51.302033 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:53.302019949 +0000 UTC m=+4.821021040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hznz" (UniqueName: "kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 23:19:52.650871 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:52.650840 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h4hmx" Oct 30 23:19:52.905922 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:52.905837 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:52.928698 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:52.928666 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:19:53.223900 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:53.223812 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:53.223900 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:53.223859 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:53.223900 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:53.223893 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:53.224171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:53.223921 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:53.224171 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.223935 7570 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 23:19:53.224171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:53.223940 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:53.224171 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.223992 7570 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 23:19:53.224171 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.224018 7570 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 23:19:53.224171 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.224017 7570 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 23:19:53.224171 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.224033 7570 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 23:19:53.224171 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.224002 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:57.223987506 +0000 UTC m=+8.742988609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 23:19:53.224171 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.224073 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:57.224055127 +0000 UTC m=+8.743056203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-config" not registered Oct 30 23:19:53.224171 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.224087 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:57.22408177 +0000 UTC m=+8.743082838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"service-ca" not registered Oct 30 23:19:53.224171 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.224109 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:57.224097077 +0000 UTC m=+8.743098146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 23:19:53.224171 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.224125 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:57.224118944 +0000 UTC m=+8.743120012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-serving-cert" not registered Oct 30 23:19:53.224598 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:53.224186 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:53.224598 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.224256 7570 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 23:19:53.224598 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.224282 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:57.224275999 +0000 UTC m=+8.743277068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-oauth-config" not registered Oct 30 23:19:53.324645 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:53.324608 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hznz\" (UniqueName: \"kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:53.324793 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.324772 7570 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 23:19:53.324829 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.324794 7570 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 23:19:53.324829 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.324805 7570 projected.go:196] Error preparing data for projected volume kube-api-access-9hznz for pod openshift-console/console-586c8ddf94-s29tc: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 23:19:53.324885 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:53.324861 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:19:57.324847822 +0000 UTC m=+8.843848911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hznz" (UniqueName: "kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 23:19:53.706748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:53.706723 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:19:55.046597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:55.046566 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:19:57.271362 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:57.271322 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:57.271372 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:57.271408 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:57.271435 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:57.271452 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.271493 7570 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:57.271513 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.271564 7570 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.271580 7570 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.271590 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:05.271570288 +0000 UTC m=+16.790571377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.271594 7570 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.271597 7570 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.271625 7570 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.271668 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:05.271621664 +0000 UTC m=+16.790622733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-serving-cert" not registered Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.271687 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:05.27167717 +0000 UTC m=+16.790678239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.271702 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:05.271692511 +0000 UTC m=+16.790693580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"service-ca" not registered Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.271715 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:05.271707896 +0000 UTC m=+16.790708965 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-config" not registered Oct 30 23:19:57.271815 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.271729 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:05.271721676 +0000 UTC m=+16.790722749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-oauth-config" not registered Oct 30 23:19:57.372766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:57.372727 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hznz\" (UniqueName: \"kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:19:57.372959 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.372903 7570 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 23:19:57.372959 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.372954 7570 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 23:19:57.373056 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.372966 7570 projected.go:196] Error preparing data for projected volume kube-api-access-9hznz for pod openshift-console/console-586c8ddf94-s29tc: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 23:19:57.373056 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:57.373024 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:05.373008631 +0000 UTC m=+16.892009699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hznz" (UniqueName: "kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 23:19:58.327968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:58.327887 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-89shx" Oct 30 23:19:58.864380 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:58.864352 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:19:58.891165 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:58.891135 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:59.361502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:59.361472 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:19:59.409978 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:19:59.409928 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:19:59.708344 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:19:59.708266 7570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:20:05.354806 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:05.354767 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:20:05.354806 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:05.354810 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.354884 7570 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.354887 7570 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.354946 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:21.354932688 +0000 UTC m=+32.873933757 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:05.354947 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.354959 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:21.354953552 +0000 UTC m=+32.873954621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:05.355019 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:05.355043 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.355102 7570 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:05.355115 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.355130 7570 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.355105 7570 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.355160 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:21.355148686 +0000 UTC m=+32.874149754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"service-ca" not registered Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.355174 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:21.355168451 +0000 UTC m=+32.874169519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-serving-cert" not registered Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.355194 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:21.355179851 +0000 UTC m=+32.874180940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-config" not registered Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.355220 7570 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 23:20:05.355286 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.355269 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:21.35526077 +0000 UTC m=+32.874261838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : object "openshift-console"/"console-oauth-config" not registered Oct 30 23:20:05.456412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:05.456370 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hznz\" (UniqueName: \"kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz\") pod \"console-586c8ddf94-s29tc\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:20:05.456627 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.456550 7570 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 23:20:05.456627 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.456566 7570 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 23:20:05.456627 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.456576 7570 projected.go:196] Error preparing data for projected volume kube-api-access-9hznz for pod openshift-console/console-586c8ddf94-s29tc: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 23:20:05.456627 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:05.456625 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz podName:b5349bf3-d267-4d03-8f45-60e58c5f881e nodeName:}" failed. No retries permitted until 2026-10-30 23:20:21.456613367 +0000 UTC m=+32.975614448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hznz" (UniqueName: "kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz") pod "console-586c8ddf94-s29tc" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 23:20:07.847842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:07.847809 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" Oct 30 23:20:12.062220 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:12.062174 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:20:14.592414 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.592366 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-586c8ddf94-s29tc" podUID="b5349bf3-d267-4d03-8f45-60e58c5f881e" containerName="console" containerID="cri-o://8c67e66da002223816af66292992c69acafb3705be263bd61e4656a7ad48f4a3" gracePeriod=15 Oct 30 23:20:14.776629 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.776593 7570 generic.go:356] "Generic (PLEG): container finished" podID="b5349bf3-d267-4d03-8f45-60e58c5f881e" containerID="8c67e66da002223816af66292992c69acafb3705be263bd61e4656a7ad48f4a3" exitCode=0 Oct 30 23:20:14.776784 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.776671 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586c8ddf94-s29tc" event={"ID":"b5349bf3-d267-4d03-8f45-60e58c5f881e","Type":"ContainerDied","Data":"8c67e66da002223816af66292992c69acafb3705be263bd61e4656a7ad48f4a3"} Oct 30 23:20:14.838729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.838702 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:20:14.953408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.953318 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert\") pod \"b5349bf3-d267-4d03-8f45-60e58c5f881e\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " Oct 30 23:20:14.953408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.953371 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5349bf3-d267-4d03-8f45-60e58c5f881e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5349bf3-d267-4d03-8f45-60e58c5f881e-tmp\") pod \"b5349bf3-d267-4d03-8f45-60e58c5f881e\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " Oct 30 23:20:14.953408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.953410 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle\") pod \"b5349bf3-d267-4d03-8f45-60e58c5f881e\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " Oct 30 23:20:14.953691 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.953500 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config\") pod \"b5349bf3-d267-4d03-8f45-60e58c5f881e\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " Oct 30 23:20:14.953691 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.953539 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca\") pod \"b5349bf3-d267-4d03-8f45-60e58c5f881e\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " Oct 30 23:20:14.953691 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.953579 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert\") pod \"b5349bf3-d267-4d03-8f45-60e58c5f881e\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " Oct 30 23:20:14.953691 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.953643 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz\" (UniqueName: \"kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz\") pod \"b5349bf3-d267-4d03-8f45-60e58c5f881e\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " Oct 30 23:20:14.953691 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.953665 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config\") pod \"b5349bf3-d267-4d03-8f45-60e58c5f881e\" (UID: \"b5349bf3-d267-4d03-8f45-60e58c5f881e\") " Oct 30 23:20:14.953901 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.953756 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5349bf3-d267-4d03-8f45-60e58c5f881e-tmp" pod "b5349bf3-d267-4d03-8f45-60e58c5f881e" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:20:14.953901 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.953785 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert" pod "b5349bf3-d267-4d03-8f45-60e58c5f881e" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:20:14.954043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.954020 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle" pod "b5349bf3-d267-4d03-8f45-60e58c5f881e" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:20:14.954043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.954036 7570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-oauth-serving-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:20:14.954154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.954057 7570 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5349bf3-d267-4d03-8f45-60e58c5f881e-tmp\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:20:14.954154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.954079 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca" pod "b5349bf3-d267-4d03-8f45-60e58c5f881e" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:20:14.954154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.954128 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config" pod "b5349bf3-d267-4d03-8f45-60e58c5f881e" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:20:14.956295 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.956268 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config" pod "b5349bf3-d267-4d03-8f45-60e58c5f881e" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:20:14.956407 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.956302 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz" pod "b5349bf3-d267-4d03-8f45-60e58c5f881e" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e"). InnerVolumeSpecName "kube-api-access-9hznz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:20:14.956407 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:14.956328 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert" pod "b5349bf3-d267-4d03-8f45-60e58c5f881e" (UID: "b5349bf3-d267-4d03-8f45-60e58c5f881e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:20:15.055014 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:15.054981 7570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-config\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:20:15.055014 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:15.055008 7570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:20:15.055014 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:15.055017 7570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-serving-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:20:15.055256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:15.055030 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hznz\" (UniqueName: \"kubernetes.io/projected/b5349bf3-d267-4d03-8f45-60e58c5f881e-kube-api-access-9hznz\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:20:15.055256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:15.055039 7570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5349bf3-d267-4d03-8f45-60e58c5f881e-console-oauth-config\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:20:15.055256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:15.055047 7570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5349bf3-d267-4d03-8f45-60e58c5f881e-trusted-ca-bundle\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:20:15.781394 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:15.781361 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586c8ddf94-s29tc" event={"ID":"b5349bf3-d267-4d03-8f45-60e58c5f881e","Type":"ContainerDied","Data":"e0a005ab3d10d39c642af76fa383d89f209ea00f95db3ba5eee5a428839c7409"} Oct 30 23:20:15.781394 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:15.781391 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586c8ddf94-s29tc" Oct 30 23:20:15.781872 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:15.781431 7570 scope.go:122] "RemoveContainer" containerID="8c67e66da002223816af66292992c69acafb3705be263bd61e4656a7ad48f4a3" Oct 30 23:20:15.802933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:15.802907 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586c8ddf94-s29tc"] Oct 30 23:20:15.806918 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:15.806899 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-586c8ddf94-s29tc"] Oct 30 23:20:16.600030 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:16.599998 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5349bf3-d267-4d03-8f45-60e58c5f881e" path="/var/lib/kubelet/pods/b5349bf3-d267-4d03-8f45-60e58c5f881e/volumes" Oct 30 23:20:20.677167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:20:20.677136 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rhm2r" Oct 30 23:20:29.410870 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:29.410829 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:20:59.411014 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:20:59.410972 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:21:12.079342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:21:12.079304 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:21:29.411969 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:21:29.411884 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:21:59.410106 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:21:59.410067 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:22:29.410279 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:22:29.410245 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:22:31.241483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.241437 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc"] Oct 30 23:22:31.242021 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.241817 7570 cpu_manager.go:414] "containerMap: removing container" podUID="b5349bf3-d267-4d03-8f45-60e58c5f881e" containerName="console" Oct 30 23:22:31.242021 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.241832 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5349bf3-d267-4d03-8f45-60e58c5f881e" containerName="console" Oct 30 23:22:31.242021 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.241921 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5349bf3-d267-4d03-8f45-60e58c5f881e" containerName="console" Oct 30 23:22:31.245047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.245031 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.307388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.307351 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc"] Oct 30 23:22:31.317787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.317759 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.317908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.317805 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.317908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.317829 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-tmp\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.318005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.317916 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-metrics-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.318073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.318045 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkcs7\" (UniqueName: \"kubernetes.io/projected/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-kube-api-access-bkcs7\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.318123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.318100 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-manifests\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.341228 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.341199 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 23:22:31.341228 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.341199 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 23:22:31.341406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.341387 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 23:22:31.341946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.341933 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 23:22:31.342831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.342813 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2mrzn\"" Oct 30 23:22:31.342883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.342853 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 23:22:31.418568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.418536 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkcs7\" (UniqueName: \"kubernetes.io/projected/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-kube-api-access-bkcs7\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.418568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.418570 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-manifests\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.418732 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.418601 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.418732 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.418644 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.418732 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.418673 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-tmp\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.418732 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.418700 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-metrics-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.419018 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.419000 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-manifests\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.419059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.419005 7570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:22:31.421172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.421144 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-tmp\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.421362 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.421347 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.421523 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.421504 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.421564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.421555 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-metrics-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.431490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.431453 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkcs7\" (UniqueName: \"kubernetes.io/projected/d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e-kube-api-access-bkcs7\") pod \"opendatahub-operator-controller-manager-6986c789f7-5bhpc\" (UID: \"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.559635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.559571 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:31.875421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.875372 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc"] Oct 30 23:22:31.880121 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:22:31.880095 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4545d5a_09c1_4bc2_8d8f_4c89a0d5b05e.slice/crio-14457f1e948b332fb09b3021d68e49d32ac3ad8377e06b10d8648de696b2d2ed WatchSource:0}: Error finding container 14457f1e948b332fb09b3021d68e49d32ac3ad8377e06b10d8648de696b2d2ed: Status 404 returned error can't find the container with id 14457f1e948b332fb09b3021d68e49d32ac3ad8377e06b10d8648de696b2d2ed Oct 30 23:22:31.881797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:31.881781 7570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:22:32.281631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:32.281551 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" event={"ID":"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e","Type":"ContainerStarted","Data":"14457f1e948b332fb09b3021d68e49d32ac3ad8377e06b10d8648de696b2d2ed"} Oct 30 23:22:34.357587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.357548 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq"] Oct 30 23:22:34.361501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.361480 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" Oct 30 23:22:34.366857 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.366834 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 23:22:34.366984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.366863 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:22:34.366984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.366918 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-x7g2z\"" Oct 30 23:22:34.373137 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.373117 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq"] Oct 30 23:22:34.445683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.445653 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpzql\" (UniqueName: \"kubernetes.io/projected/2b47b3cc-9735-45f1-8d87-7edd6ef95dd8-kube-api-access-dpzql\") pod \"jobset-operator-674cd59dc5-x6pgq\" (UID: \"2b47b3cc-9735-45f1-8d87-7edd6ef95dd8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" Oct 30 23:22:34.445859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.445752 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b47b3cc-9735-45f1-8d87-7edd6ef95dd8-tmp\") pod \"jobset-operator-674cd59dc5-x6pgq\" (UID: \"2b47b3cc-9735-45f1-8d87-7edd6ef95dd8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" Oct 30 23:22:34.546917 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.546888 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpzql\" (UniqueName: \"kubernetes.io/projected/2b47b3cc-9735-45f1-8d87-7edd6ef95dd8-kube-api-access-dpzql\") pod \"jobset-operator-674cd59dc5-x6pgq\" (UID: \"2b47b3cc-9735-45f1-8d87-7edd6ef95dd8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" Oct 30 23:22:34.547096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.546973 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b47b3cc-9735-45f1-8d87-7edd6ef95dd8-tmp\") pod \"jobset-operator-674cd59dc5-x6pgq\" (UID: \"2b47b3cc-9735-45f1-8d87-7edd6ef95dd8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" Oct 30 23:22:34.547428 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.547411 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b47b3cc-9735-45f1-8d87-7edd6ef95dd8-tmp\") pod \"jobset-operator-674cd59dc5-x6pgq\" (UID: \"2b47b3cc-9735-45f1-8d87-7edd6ef95dd8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" Oct 30 23:22:34.559878 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.559859 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpzql\" (UniqueName: \"kubernetes.io/projected/2b47b3cc-9735-45f1-8d87-7edd6ef95dd8-kube-api-access-dpzql\") pod \"jobset-operator-674cd59dc5-x6pgq\" (UID: \"2b47b3cc-9735-45f1-8d87-7edd6ef95dd8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" Oct 30 23:22:34.685330 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.685255 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" Oct 30 23:22:34.986035 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:22:34.986011 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b47b3cc_9735_45f1_8d87_7edd6ef95dd8.slice/crio-1128148534c3044c39d82e2db1d7f10c97c2b256cb96df1ffa3044c86250c0c3 WatchSource:0}: Error finding container 1128148534c3044c39d82e2db1d7f10c97c2b256cb96df1ffa3044c86250c0c3: Status 404 returned error can't find the container with id 1128148534c3044c39d82e2db1d7f10c97c2b256cb96df1ffa3044c86250c0c3 Oct 30 23:22:34.989389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:34.989366 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq"] Oct 30 23:22:35.298115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:35.298040 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" event={"ID":"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e","Type":"ContainerStarted","Data":"ad7d8a68b59e0f2c60fc4904638b1f18c64c62292fecd9d61a3011c28ba7e541"} Oct 30 23:22:35.299279 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:35.299254 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" event={"ID":"2b47b3cc-9735-45f1-8d87-7edd6ef95dd8","Type":"ContainerStarted","Data":"1128148534c3044c39d82e2db1d7f10c97c2b256cb96df1ffa3044c86250c0c3"} Oct 30 23:22:36.305189 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:36.305140 7570 generic.go:356] "Generic (PLEG): container finished" podID="d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e" containerID="ad7d8a68b59e0f2c60fc4904638b1f18c64c62292fecd9d61a3011c28ba7e541" exitCode=0 Oct 30 23:22:36.305606 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:36.305213 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" event={"ID":"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e","Type":"ContainerDied","Data":"ad7d8a68b59e0f2c60fc4904638b1f18c64c62292fecd9d61a3011c28ba7e541"} Oct 30 23:22:37.311111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:37.311072 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" event={"ID":"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e","Type":"ContainerStarted","Data":"c5562a3342bf0ed100c461b7305c75c61de2b03f84d5c91d893824188f07732f"} Oct 30 23:22:37.311544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:37.311190 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:37.374801 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:37.374755 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" podStartSLOduration=2.346436196 podStartE2EDuration="6.374742716s" totalImagesPullingTime="3.518002803s" totalInitContainerRuntime="510.303717ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:37.373203426 +0000 UTC m=+168.892204553" watchObservedRunningTime="2026-10-30 23:22:37.374742716 +0000 UTC m=+168.893743799" Oct 30 23:22:39.323325 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:39.323287 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" event={"ID":"2b47b3cc-9735-45f1-8d87-7edd6ef95dd8","Type":"ContainerStarted","Data":"e744dd43c6e82155215305ded5379603f8103350e6f2fb7975b8e476b2e81b76"} Oct 30 23:22:39.345727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:39.345680 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" podStartSLOduration=1.152627611 podStartE2EDuration="5.34566258s" totalImagesPullingTime="4.193034969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:39.344732855 +0000 UTC m=+170.863733946" watchObservedRunningTime="2026-10-30 23:22:39.34566258 +0000 UTC m=+170.864663671" Oct 30 23:22:48.317937 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:48.317905 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:22:51.787381 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:51.787343 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd"] Oct 30 23:22:51.797671 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:51.797644 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:22:51.804306 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:51.804151 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd"] Oct 30 23:22:51.813832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:51.813815 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 23:22:51.813946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:51.813840 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 23:22:51.813946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:51.813880 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fl65j\"" Oct 30 23:22:51.897599 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:51.897571 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/baaa353f-43a5-43e5-a73c-92d1840ebdae-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vnmqd\" (UID: \"baaa353f-43a5-43e5-a73c-92d1840ebdae\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:22:51.897768 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:51.897745 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqd2g\" (UniqueName: \"kubernetes.io/projected/baaa353f-43a5-43e5-a73c-92d1840ebdae-kube-api-access-jqd2g\") pod \"cert-manager-webhook-5dbbbf75cc-vnmqd\" (UID: \"baaa353f-43a5-43e5-a73c-92d1840ebdae\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:22:51.998971 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:51.998939 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqd2g\" (UniqueName: \"kubernetes.io/projected/baaa353f-43a5-43e5-a73c-92d1840ebdae-kube-api-access-jqd2g\") pod \"cert-manager-webhook-5dbbbf75cc-vnmqd\" (UID: \"baaa353f-43a5-43e5-a73c-92d1840ebdae\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:22:51.999100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:51.998984 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/baaa353f-43a5-43e5-a73c-92d1840ebdae-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vnmqd\" (UID: \"baaa353f-43a5-43e5-a73c-92d1840ebdae\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:22:52.029550 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:52.029520 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/baaa353f-43a5-43e5-a73c-92d1840ebdae-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vnmqd\" (UID: \"baaa353f-43a5-43e5-a73c-92d1840ebdae\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:22:52.037506 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:52.037440 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqd2g\" (UniqueName: \"kubernetes.io/projected/baaa353f-43a5-43e5-a73c-92d1840ebdae-kube-api-access-jqd2g\") pod \"cert-manager-webhook-5dbbbf75cc-vnmqd\" (UID: \"baaa353f-43a5-43e5-a73c-92d1840ebdae\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:22:52.110046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:52.110021 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:22:52.437315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:52.437287 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd"] Oct 30 23:22:52.439801 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:22:52.439770 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaaa353f_43a5_43e5_a73c_92d1840ebdae.slice/crio-ca36eb912ff44ed68f49ab517ac752d10a01a10976620acb9ecf233ff9f992f7 WatchSource:0}: Error finding container ca36eb912ff44ed68f49ab517ac752d10a01a10976620acb9ecf233ff9f992f7: Status 404 returned error can't find the container with id ca36eb912ff44ed68f49ab517ac752d10a01a10976620acb9ecf233ff9f992f7 Oct 30 23:22:53.378369 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:53.378334 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" event={"ID":"baaa353f-43a5-43e5-a73c-92d1840ebdae","Type":"ContainerStarted","Data":"ca36eb912ff44ed68f49ab517ac752d10a01a10976620acb9ecf233ff9f992f7"} Oct 30 23:22:59.245442 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:59.245366 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m"] Oct 30 23:22:59.248744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:59.248728 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" Oct 30 23:22:59.262305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:59.262283 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m"] Oct 30 23:22:59.262952 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:59.262933 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5f557\"" Oct 30 23:22:59.364621 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:59.364590 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/347b9a5b-729f-419a-bce5-749ac4aca5b5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-68r7m\" (UID: \"347b9a5b-729f-419a-bce5-749ac4aca5b5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" Oct 30 23:22:59.364786 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:59.364698 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzml8\" (UniqueName: \"kubernetes.io/projected/347b9a5b-729f-419a-bce5-749ac4aca5b5-kube-api-access-lzml8\") pod \"cert-manager-cainjector-5c57bbd4fc-68r7m\" (UID: \"347b9a5b-729f-419a-bce5-749ac4aca5b5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" Oct 30 23:22:59.410506 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:22:59.410440 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:22:59.465560 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:59.465529 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzml8\" (UniqueName: \"kubernetes.io/projected/347b9a5b-729f-419a-bce5-749ac4aca5b5-kube-api-access-lzml8\") pod \"cert-manager-cainjector-5c57bbd4fc-68r7m\" (UID: \"347b9a5b-729f-419a-bce5-749ac4aca5b5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" Oct 30 23:22:59.465723 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:59.465605 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/347b9a5b-729f-419a-bce5-749ac4aca5b5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-68r7m\" (UID: \"347b9a5b-729f-419a-bce5-749ac4aca5b5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" Oct 30 23:22:59.481565 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:59.481527 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzml8\" (UniqueName: \"kubernetes.io/projected/347b9a5b-729f-419a-bce5-749ac4aca5b5-kube-api-access-lzml8\") pod \"cert-manager-cainjector-5c57bbd4fc-68r7m\" (UID: \"347b9a5b-729f-419a-bce5-749ac4aca5b5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" Oct 30 23:22:59.482132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:59.482109 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/347b9a5b-729f-419a-bce5-749ac4aca5b5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-68r7m\" (UID: \"347b9a5b-729f-419a-bce5-749ac4aca5b5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" Oct 30 23:22:59.560893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:59.560822 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" Oct 30 23:22:59.836552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:22:59.836520 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m"] Oct 30 23:22:59.839710 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:22:59.839676 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod347b9a5b_729f_419a_bce5_749ac4aca5b5.slice/crio-48f657194580a58c6888b00d96192a5007c313be61d3368d05b384bcc5ce4068 WatchSource:0}: Error finding container 48f657194580a58c6888b00d96192a5007c313be61d3368d05b384bcc5ce4068: Status 404 returned error can't find the container with id 48f657194580a58c6888b00d96192a5007c313be61d3368d05b384bcc5ce4068 Oct 30 23:23:00.376520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.376491 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh"] Oct 30 23:23:00.380992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.380974 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" Oct 30 23:23:00.399242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.399215 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 23:23:00.399242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.399220 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 23:23:00.399516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.399502 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 23:23:00.399651 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.399638 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ttclb\"" Oct 30 23:23:00.400420 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.400409 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 23:23:00.407441 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.407414 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" event={"ID":"347b9a5b-729f-419a-bce5-749ac4aca5b5","Type":"ContainerStarted","Data":"48f657194580a58c6888b00d96192a5007c313be61d3368d05b384bcc5ce4068"} Oct 30 23:23:00.453584 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.453560 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh"] Oct 30 23:23:00.474235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.474205 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b7be075-e8bb-4db1-bfab-782276c29b6e-tmp\") pod \"kube-auth-proxy-dfd47b86c-gjwxh\" (UID: \"2b7be075-e8bb-4db1-bfab-782276c29b6e\") " pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" Oct 30 23:23:00.474403 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.474249 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7be075-e8bb-4db1-bfab-782276c29b6e-tls-certs\") pod \"kube-auth-proxy-dfd47b86c-gjwxh\" (UID: \"2b7be075-e8bb-4db1-bfab-782276c29b6e\") " pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" Oct 30 23:23:00.474403 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.474375 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqd24\" (UniqueName: \"kubernetes.io/projected/2b7be075-e8bb-4db1-bfab-782276c29b6e-kube-api-access-mqd24\") pod \"kube-auth-proxy-dfd47b86c-gjwxh\" (UID: \"2b7be075-e8bb-4db1-bfab-782276c29b6e\") " pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" Oct 30 23:23:00.575870 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.575836 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqd24\" (UniqueName: \"kubernetes.io/projected/2b7be075-e8bb-4db1-bfab-782276c29b6e-kube-api-access-mqd24\") pod \"kube-auth-proxy-dfd47b86c-gjwxh\" (UID: \"2b7be075-e8bb-4db1-bfab-782276c29b6e\") " pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" Oct 30 23:23:00.576046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.575932 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b7be075-e8bb-4db1-bfab-782276c29b6e-tmp\") pod \"kube-auth-proxy-dfd47b86c-gjwxh\" (UID: \"2b7be075-e8bb-4db1-bfab-782276c29b6e\") " pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" Oct 30 23:23:00.576046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.575961 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7be075-e8bb-4db1-bfab-782276c29b6e-tls-certs\") pod \"kube-auth-proxy-dfd47b86c-gjwxh\" (UID: \"2b7be075-e8bb-4db1-bfab-782276c29b6e\") " pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" Oct 30 23:23:00.578550 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.578524 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b7be075-e8bb-4db1-bfab-782276c29b6e-tmp\") pod \"kube-auth-proxy-dfd47b86c-gjwxh\" (UID: \"2b7be075-e8bb-4db1-bfab-782276c29b6e\") " pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" Oct 30 23:23:00.578881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.578854 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7be075-e8bb-4db1-bfab-782276c29b6e-tls-certs\") pod \"kube-auth-proxy-dfd47b86c-gjwxh\" (UID: \"2b7be075-e8bb-4db1-bfab-782276c29b6e\") " pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" Oct 30 23:23:00.594604 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.594576 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqd24\" (UniqueName: \"kubernetes.io/projected/2b7be075-e8bb-4db1-bfab-782276c29b6e-kube-api-access-mqd24\") pod \"kube-auth-proxy-dfd47b86c-gjwxh\" (UID: \"2b7be075-e8bb-4db1-bfab-782276c29b6e\") " pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" Oct 30 23:23:00.693277 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.693199 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" Oct 30 23:23:00.998696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:00.998612 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh"] Oct 30 23:23:01.003239 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:23:01.003206 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7be075_e8bb_4db1_bfab_782276c29b6e.slice/crio-b61c3e41a4c2d4ebbd6b499f4699f2c6072ad4183384d66b6fb5a55c6f0d443c WatchSource:0}: Error finding container b61c3e41a4c2d4ebbd6b499f4699f2c6072ad4183384d66b6fb5a55c6f0d443c: Status 404 returned error can't find the container with id b61c3e41a4c2d4ebbd6b499f4699f2c6072ad4183384d66b6fb5a55c6f0d443c Oct 30 23:23:01.413744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:01.413712 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" event={"ID":"2b7be075-e8bb-4db1-bfab-782276c29b6e","Type":"ContainerStarted","Data":"b61c3e41a4c2d4ebbd6b499f4699f2c6072ad4183384d66b6fb5a55c6f0d443c"} Oct 30 23:23:02.421290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.420520 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" event={"ID":"baaa353f-43a5-43e5-a73c-92d1840ebdae","Type":"ContainerStarted","Data":"a3d1da19d83c6d20abbe7767b9039dd323558307503a1344bb47b92377be204d"} Oct 30 23:23:02.421290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.421247 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:23:02.460526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.460443 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podStartSLOduration=2.409387706 podStartE2EDuration="11.460423178s" totalImagesPullingTime="9.051035472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:02.458183391 +0000 UTC m=+193.977184483" watchObservedRunningTime="2026-10-30 23:23:02.460423178 +0000 UTC m=+193.979424269" Oct 30 23:23:02.805899 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.804202 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j"] Oct 30 23:23:02.813390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.813361 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:02.823361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.822192 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-lc8kv\"" Oct 30 23:23:02.824912 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.824851 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j"] Oct 30 23:23:02.853831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.853799 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 23:23:02.854314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.854292 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 23:23:02.856342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.856298 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 23:23:02.907267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.907235 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20fc3e5f-362e-41f7-bb52-2b171d7af818-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5n89j\" (UID: \"20fc3e5f-362e-41f7-bb52-2b171d7af818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:02.907442 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.907320 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20fc3e5f-362e-41f7-bb52-2b171d7af818-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5n89j\" (UID: \"20fc3e5f-362e-41f7-bb52-2b171d7af818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:02.907442 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.907430 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/20fc3e5f-362e-41f7-bb52-2b171d7af818-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5n89j\" (UID: \"20fc3e5f-362e-41f7-bb52-2b171d7af818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:02.907609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:02.907493 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wqsd\" (UniqueName: \"kubernetes.io/projected/20fc3e5f-362e-41f7-bb52-2b171d7af818-kube-api-access-8wqsd\") pod \"kserve-module-controller-manager-7566cf5fc8-5n89j\" (UID: \"20fc3e5f-362e-41f7-bb52-2b171d7af818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:03.008423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.008381 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20fc3e5f-362e-41f7-bb52-2b171d7af818-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5n89j\" (UID: \"20fc3e5f-362e-41f7-bb52-2b171d7af818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:03.008423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.008435 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20fc3e5f-362e-41f7-bb52-2b171d7af818-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5n89j\" (UID: \"20fc3e5f-362e-41f7-bb52-2b171d7af818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:03.008692 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.008501 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/20fc3e5f-362e-41f7-bb52-2b171d7af818-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5n89j\" (UID: \"20fc3e5f-362e-41f7-bb52-2b171d7af818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:03.008692 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.008533 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wqsd\" (UniqueName: \"kubernetes.io/projected/20fc3e5f-362e-41f7-bb52-2b171d7af818-kube-api-access-8wqsd\") pod \"kserve-module-controller-manager-7566cf5fc8-5n89j\" (UID: \"20fc3e5f-362e-41f7-bb52-2b171d7af818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:03.009661 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.009309 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/20fc3e5f-362e-41f7-bb52-2b171d7af818-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5n89j\" (UID: \"20fc3e5f-362e-41f7-bb52-2b171d7af818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:03.009661 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.009625 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20fc3e5f-362e-41f7-bb52-2b171d7af818-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5n89j\" (UID: \"20fc3e5f-362e-41f7-bb52-2b171d7af818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:03.009916 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.009846 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20fc3e5f-362e-41f7-bb52-2b171d7af818-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5n89j\" (UID: \"20fc3e5f-362e-41f7-bb52-2b171d7af818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:03.030438 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.030410 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wqsd\" (UniqueName: \"kubernetes.io/projected/20fc3e5f-362e-41f7-bb52-2b171d7af818-kube-api-access-8wqsd\") pod \"kserve-module-controller-manager-7566cf5fc8-5n89j\" (UID: \"20fc3e5f-362e-41f7-bb52-2b171d7af818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:03.100276 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.100243 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-p5xwl"] Oct 30 23:23:03.105354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.105331 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-p5xwl" Oct 30 23:23:03.118490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.115643 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-n7mcg\"" Oct 30 23:23:03.118490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.116541 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-p5xwl"] Oct 30 23:23:03.131884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.131859 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:03.215187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.214943 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee540c2e-f028-44ee-be53-d178b286fb02-bound-sa-token\") pod \"cert-manager-597b95cb84-p5xwl\" (UID: \"ee540c2e-f028-44ee-be53-d178b286fb02\") " pod="cert-manager/cert-manager-597b95cb84-p5xwl" Oct 30 23:23:03.215187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.215084 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqck\" (UniqueName: \"kubernetes.io/projected/ee540c2e-f028-44ee-be53-d178b286fb02-kube-api-access-mvqck\") pod \"cert-manager-597b95cb84-p5xwl\" (UID: \"ee540c2e-f028-44ee-be53-d178b286fb02\") " pod="cert-manager/cert-manager-597b95cb84-p5xwl" Oct 30 23:23:03.317344 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.316337 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee540c2e-f028-44ee-be53-d178b286fb02-bound-sa-token\") pod \"cert-manager-597b95cb84-p5xwl\" (UID: \"ee540c2e-f028-44ee-be53-d178b286fb02\") " pod="cert-manager/cert-manager-597b95cb84-p5xwl" Oct 30 23:23:03.317344 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.316536 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqck\" (UniqueName: \"kubernetes.io/projected/ee540c2e-f028-44ee-be53-d178b286fb02-kube-api-access-mvqck\") pod \"cert-manager-597b95cb84-p5xwl\" (UID: \"ee540c2e-f028-44ee-be53-d178b286fb02\") " pod="cert-manager/cert-manager-597b95cb84-p5xwl" Oct 30 23:23:03.330139 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.330115 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee540c2e-f028-44ee-be53-d178b286fb02-bound-sa-token\") pod \"cert-manager-597b95cb84-p5xwl\" (UID: \"ee540c2e-f028-44ee-be53-d178b286fb02\") " pod="cert-manager/cert-manager-597b95cb84-p5xwl" Oct 30 23:23:03.330798 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.330773 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqck\" (UniqueName: \"kubernetes.io/projected/ee540c2e-f028-44ee-be53-d178b286fb02-kube-api-access-mvqck\") pod \"cert-manager-597b95cb84-p5xwl\" (UID: \"ee540c2e-f028-44ee-be53-d178b286fb02\") " pod="cert-manager/cert-manager-597b95cb84-p5xwl" Oct 30 23:23:03.422216 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.422191 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-p5xwl" Oct 30 23:23:03.427496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.427452 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j"] Oct 30 23:23:03.434835 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:23:03.434813 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fc3e5f_362e_41f7_bb52_2b171d7af818.slice/crio-7420fd246174ebd87947c782e8f7219efd8c94c179a04f09cd740aaf6e3b2200 WatchSource:0}: Error finding container 7420fd246174ebd87947c782e8f7219efd8c94c179a04f09cd740aaf6e3b2200: Status 404 returned error can't find the container with id 7420fd246174ebd87947c782e8f7219efd8c94c179a04f09cd740aaf6e3b2200 Oct 30 23:23:03.701428 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:03.701353 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-p5xwl"] Oct 30 23:23:03.710350 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:23:03.710319 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee540c2e_f028_44ee_be53_d178b286fb02.slice/crio-4ee8b39ac66bbe4f62ffaae673d6b544cd8c0a517d1de81de70fa4b0eb85450e WatchSource:0}: Error finding container 4ee8b39ac66bbe4f62ffaae673d6b544cd8c0a517d1de81de70fa4b0eb85450e: Status 404 returned error can't find the container with id 4ee8b39ac66bbe4f62ffaae673d6b544cd8c0a517d1de81de70fa4b0eb85450e Oct 30 23:23:04.431149 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:04.431113 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-p5xwl" event={"ID":"ee540c2e-f028-44ee-be53-d178b286fb02","Type":"ContainerStarted","Data":"318dfc9987ebf2f548bc401821e12fd61ce5837c82803f1b0ead37e89b9a688b"} Oct 30 23:23:04.431619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:04.431160 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-p5xwl" event={"ID":"ee540c2e-f028-44ee-be53-d178b286fb02","Type":"ContainerStarted","Data":"4ee8b39ac66bbe4f62ffaae673d6b544cd8c0a517d1de81de70fa4b0eb85450e"} Oct 30 23:23:04.432741 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:04.432698 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" event={"ID":"2b7be075-e8bb-4db1-bfab-782276c29b6e","Type":"ContainerStarted","Data":"85e9bc68d5b82f554d24af1226eb8a86a1ae76ceef30d8a897d1aa5ab1e2c875"} Oct 30 23:23:04.434120 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:04.434096 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" event={"ID":"347b9a5b-729f-419a-bce5-749ac4aca5b5","Type":"ContainerStarted","Data":"227922d5b20c1762b1662970a8dc5aa42f92fd5e306aca4c173a37e02e3feb31"} Oct 30 23:23:04.435193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:04.435171 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" event={"ID":"20fc3e5f-362e-41f7-bb52-2b171d7af818","Type":"ContainerStarted","Data":"7420fd246174ebd87947c782e8f7219efd8c94c179a04f09cd740aaf6e3b2200"} Oct 30 23:23:04.452998 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:04.452654 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-p5xwl" podStartSLOduration=1.452641632 podStartE2EDuration="1.452641632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:04.451727036 +0000 UTC m=+195.970728138" watchObservedRunningTime="2026-10-30 23:23:04.452641632 +0000 UTC m=+195.971642726" Oct 30 23:23:04.474433 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:04.474381 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" podStartSLOduration=1.931024019 podStartE2EDuration="5.474364349s" totalImagesPullingTime="3.54334033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:04.473217753 +0000 UTC m=+195.992218844" watchObservedRunningTime="2026-10-30 23:23:04.474364349 +0000 UTC m=+195.993365440" Oct 30 23:23:04.494097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:04.494057 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-dfd47b86c-gjwxh" podStartSLOduration=1.585493144 podStartE2EDuration="4.494043901s" totalImagesPullingTime="2.908550757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:04.492420561 +0000 UTC m=+196.011421650" watchObservedRunningTime="2026-10-30 23:23:04.494043901 +0000 UTC m=+196.013044985" Oct 30 23:23:06.446028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:06.445994 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" event={"ID":"20fc3e5f-362e-41f7-bb52-2b171d7af818","Type":"ContainerStarted","Data":"086b42ff31df3806fabe032ac13c48753bc1001a7c213cbaa603d1d1f2bdd8fb"} Oct 30 23:23:06.446452 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:06.446151 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:06.469288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:06.469248 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podStartSLOduration=2.06043903 podStartE2EDuration="4.469234448s" totalImagesPullingTime="2.408795418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:06.467963935 +0000 UTC m=+197.986965024" watchObservedRunningTime="2026-10-30 23:23:06.469234448 +0000 UTC m=+197.988235537" Oct 30 23:23:09.437635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:09.437602 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:23:27.451335 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:27.451297 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:23:29.411406 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:23:29.411369 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:23:56.892359 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:56.892327 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-4blbf"] Oct 30 23:23:56.901727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:56.901711 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:23:56.908070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:56.908047 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-4blbf"] Oct 30 23:23:56.930593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:56.930553 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 23:23:56.930759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:56.930609 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-vf4pc\"" Oct 30 23:23:56.931036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:56.931021 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 23:23:57.006820 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:57.006796 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1793bf9b-92db-4dae-bfd6-f86e5b076739-cert\") pod \"odh-model-controller-769875db56-4blbf\" (UID: \"1793bf9b-92db-4dae-bfd6-f86e5b076739\") " pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:23:57.006967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:57.006829 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1793bf9b-92db-4dae-bfd6-f86e5b076739-metrics-certs\") pod \"odh-model-controller-769875db56-4blbf\" (UID: \"1793bf9b-92db-4dae-bfd6-f86e5b076739\") " pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:23:57.006967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:57.006853 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbs5\" (UniqueName: \"kubernetes.io/projected/1793bf9b-92db-4dae-bfd6-f86e5b076739-kube-api-access-nlbs5\") pod \"odh-model-controller-769875db56-4blbf\" (UID: \"1793bf9b-92db-4dae-bfd6-f86e5b076739\") " pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:23:57.108281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:57.108252 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1793bf9b-92db-4dae-bfd6-f86e5b076739-cert\") pod \"odh-model-controller-769875db56-4blbf\" (UID: \"1793bf9b-92db-4dae-bfd6-f86e5b076739\") " pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:23:57.108439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:57.108285 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1793bf9b-92db-4dae-bfd6-f86e5b076739-metrics-certs\") pod \"odh-model-controller-769875db56-4blbf\" (UID: \"1793bf9b-92db-4dae-bfd6-f86e5b076739\") " pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:23:57.108439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:57.108310 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbs5\" (UniqueName: \"kubernetes.io/projected/1793bf9b-92db-4dae-bfd6-f86e5b076739-kube-api-access-nlbs5\") pod \"odh-model-controller-769875db56-4blbf\" (UID: \"1793bf9b-92db-4dae-bfd6-f86e5b076739\") " pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:23:57.110914 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:57.110894 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1793bf9b-92db-4dae-bfd6-f86e5b076739-metrics-certs\") pod \"odh-model-controller-769875db56-4blbf\" (UID: \"1793bf9b-92db-4dae-bfd6-f86e5b076739\") " pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:23:57.111013 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:57.110922 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1793bf9b-92db-4dae-bfd6-f86e5b076739-cert\") pod \"odh-model-controller-769875db56-4blbf\" (UID: \"1793bf9b-92db-4dae-bfd6-f86e5b076739\") " pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:23:57.120911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:57.120888 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbs5\" (UniqueName: \"kubernetes.io/projected/1793bf9b-92db-4dae-bfd6-f86e5b076739-kube-api-access-nlbs5\") pod \"odh-model-controller-769875db56-4blbf\" (UID: \"1793bf9b-92db-4dae-bfd6-f86e5b076739\") " pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:23:57.216013 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:57.215938 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:23:57.490969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:57.490896 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-4blbf"] Oct 30 23:23:57.493799 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:23:57.493765 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1793bf9b_92db_4dae_bfd6_f86e5b076739.slice/crio-8bd775f2b9bb04c747c7d5ad2b978fd4abb10c1916fe91a52fd46c3a99c6796a WatchSource:0}: Error finding container 8bd775f2b9bb04c747c7d5ad2b978fd4abb10c1916fe91a52fd46c3a99c6796a: Status 404 returned error can't find the container with id 8bd775f2b9bb04c747c7d5ad2b978fd4abb10c1916fe91a52fd46c3a99c6796a Oct 30 23:23:57.679637 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:23:57.679606 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-4blbf" event={"ID":"1793bf9b-92db-4dae-bfd6-f86e5b076739","Type":"ContainerStarted","Data":"8bd775f2b9bb04c747c7d5ad2b978fd4abb10c1916fe91a52fd46c3a99c6796a"} Oct 30 23:23:59.419718 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:23:59.419674 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:24:00.695631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:00.695595 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-4blbf" event={"ID":"1793bf9b-92db-4dae-bfd6-f86e5b076739","Type":"ContainerStarted","Data":"190fd53e5d4cc3e3087e5f2daddbdd346deb07e3206ffa107d2a1d42b517c394"} Oct 30 23:24:00.696023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:00.695737 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:24:00.720104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:00.720055 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-4blbf" podStartSLOduration=2.096982098 podStartE2EDuration="4.720042407s" totalImagesPullingTime="2.623060309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:00.718893383 +0000 UTC m=+252.237894474" watchObservedRunningTime="2026-10-30 23:24:00.720042407 +0000 UTC m=+252.239043494" Oct 30 23:24:06.647821 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.647788 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk"] Oct 30 23:24:06.651141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.651120 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.656306 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.656277 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 23:24:06.656431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.656307 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 23:24:06.656431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.656379 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-2q87t\"" Oct 30 23:24:06.656431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.656400 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 23:24:06.660246 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.660221 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk"] Oct 30 23:24:06.797906 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.797869 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f7c5040c-6dbc-424d-a258-94092e75a027-manager-config\") pod \"jobset-controller-manager-57b79bd74c-mqnfk\" (UID: \"f7c5040c-6dbc-424d-a258-94092e75a027\") " pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.798080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.797936 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7c5040c-6dbc-424d-a258-94092e75a027-cert\") pod \"jobset-controller-manager-57b79bd74c-mqnfk\" (UID: \"f7c5040c-6dbc-424d-a258-94092e75a027\") " pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.798080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.797998 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcgc4\" (UniqueName: \"kubernetes.io/projected/f7c5040c-6dbc-424d-a258-94092e75a027-kube-api-access-xcgc4\") pod \"jobset-controller-manager-57b79bd74c-mqnfk\" (UID: \"f7c5040c-6dbc-424d-a258-94092e75a027\") " pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.798080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.798066 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c5040c-6dbc-424d-a258-94092e75a027-metrics-certs\") pod \"jobset-controller-manager-57b79bd74c-mqnfk\" (UID: \"f7c5040c-6dbc-424d-a258-94092e75a027\") " pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.898884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.898806 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcgc4\" (UniqueName: \"kubernetes.io/projected/f7c5040c-6dbc-424d-a258-94092e75a027-kube-api-access-xcgc4\") pod \"jobset-controller-manager-57b79bd74c-mqnfk\" (UID: \"f7c5040c-6dbc-424d-a258-94092e75a027\") " pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.898884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.898851 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c5040c-6dbc-424d-a258-94092e75a027-metrics-certs\") pod \"jobset-controller-manager-57b79bd74c-mqnfk\" (UID: \"f7c5040c-6dbc-424d-a258-94092e75a027\") " pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.899106 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.898903 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f7c5040c-6dbc-424d-a258-94092e75a027-manager-config\") pod \"jobset-controller-manager-57b79bd74c-mqnfk\" (UID: \"f7c5040c-6dbc-424d-a258-94092e75a027\") " pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.899106 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.898943 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7c5040c-6dbc-424d-a258-94092e75a027-cert\") pod \"jobset-controller-manager-57b79bd74c-mqnfk\" (UID: \"f7c5040c-6dbc-424d-a258-94092e75a027\") " pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.899889 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.899850 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f7c5040c-6dbc-424d-a258-94092e75a027-manager-config\") pod \"jobset-controller-manager-57b79bd74c-mqnfk\" (UID: \"f7c5040c-6dbc-424d-a258-94092e75a027\") " pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.901549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.901526 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7c5040c-6dbc-424d-a258-94092e75a027-cert\") pod \"jobset-controller-manager-57b79bd74c-mqnfk\" (UID: \"f7c5040c-6dbc-424d-a258-94092e75a027\") " pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.901711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.901688 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c5040c-6dbc-424d-a258-94092e75a027-metrics-certs\") pod \"jobset-controller-manager-57b79bd74c-mqnfk\" (UID: \"f7c5040c-6dbc-424d-a258-94092e75a027\") " pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.911648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.911622 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcgc4\" (UniqueName: \"kubernetes.io/projected/f7c5040c-6dbc-424d-a258-94092e75a027-kube-api-access-xcgc4\") pod \"jobset-controller-manager-57b79bd74c-mqnfk\" (UID: \"f7c5040c-6dbc-424d-a258-94092e75a027\") " pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:06.963540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:06.963501 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:07.244405 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:07.244283 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk"] Oct 30 23:24:07.247127 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:24:07.247096 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c5040c_6dbc_424d_a258_94092e75a027.slice/crio-8c6ffb1a0c0ae2fea8866b6f184f463715abec7f450889f34e764499248b5770 WatchSource:0}: Error finding container 8c6ffb1a0c0ae2fea8866b6f184f463715abec7f450889f34e764499248b5770: Status 404 returned error can't find the container with id 8c6ffb1a0c0ae2fea8866b6f184f463715abec7f450889f34e764499248b5770 Oct 30 23:24:07.728607 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:07.728568 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" event={"ID":"f7c5040c-6dbc-424d-a258-94092e75a027","Type":"ContainerStarted","Data":"8c6ffb1a0c0ae2fea8866b6f184f463715abec7f450889f34e764499248b5770"} Oct 30 23:24:11.702164 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:11.702132 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:24:11.748902 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:11.748865 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" event={"ID":"f7c5040c-6dbc-424d-a258-94092e75a027","Type":"ContainerStarted","Data":"d8a0528c523be2bba1198e2108c245b96b5da75b4c7665eb9f0ee23394be4482"} Oct 30 23:24:11.749077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:11.749010 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:11.767505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:11.767423 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" podStartSLOduration=1.8404191380000001 podStartE2EDuration="5.767407094s" totalImagesPullingTime="3.926987956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:11.7671013 +0000 UTC m=+263.286102405" watchObservedRunningTime="2026-10-30 23:24:11.767407094 +0000 UTC m=+263.286408186" Oct 30 23:24:22.758907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:24:22.758878 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:24:29.412036 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:24:29.411959 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:24:59.411602 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:24:59.411557 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:25:29.412089 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:25:29.412051 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:25:59.411071 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:25:59.410986 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:26:29.411895 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:26:29.411856 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:26:59.411995 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:26:59.411959 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:27:29.410652 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:27:29.410575 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:27:59.410979 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:27:59.410940 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:28:29.412234 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:28:29.412197 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:28:59.411919 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:28:59.411838 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:29:29.412601 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:29:29.412560 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:29:59.409841 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:29:59.409805 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:30:29.411436 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:30:29.411348 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:30:59.410381 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:30:59.410345 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:31:29.412312 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:31:29.412273 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:31:59.409766 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:31:59.409676 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:32:29.410161 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:32:29.410126 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:32:59.411034 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:32:59.410996 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:33:29.410107 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:33:29.410033 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:33:42.059644 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:42.059611 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pswg7"] Oct 30 23:33:42.063378 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:42.063355 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" Oct 30 23:33:42.076826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:42.076805 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pswg7"] Oct 30 23:33:42.078520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:42.078494 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 23:33:42.078620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:42.078571 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 23:33:42.078667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:42.078620 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hc9wr\"" Oct 30 23:33:42.206192 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:42.206171 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxrhs\" (UniqueName: \"kubernetes.io/projected/bc55b446-6292-4791-a9eb-373860d01f32-kube-api-access-wxrhs\") pod \"authorino-56c6c9ddd5-pswg7\" (UID: \"bc55b446-6292-4791-a9eb-373860d01f32\") " pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" Oct 30 23:33:42.307131 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:42.307103 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxrhs\" (UniqueName: \"kubernetes.io/projected/bc55b446-6292-4791-a9eb-373860d01f32-kube-api-access-wxrhs\") pod \"authorino-56c6c9ddd5-pswg7\" (UID: \"bc55b446-6292-4791-a9eb-373860d01f32\") " pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" Oct 30 23:33:42.320004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:42.319925 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxrhs\" (UniqueName: \"kubernetes.io/projected/bc55b446-6292-4791-a9eb-373860d01f32-kube-api-access-wxrhs\") pod \"authorino-56c6c9ddd5-pswg7\" (UID: \"bc55b446-6292-4791-a9eb-373860d01f32\") " pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" Oct 30 23:33:42.374747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:42.374721 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" Oct 30 23:33:42.646172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:42.646149 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pswg7"] Oct 30 23:33:42.648749 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:33:42.648718 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc55b446_6292_4791_a9eb_373860d01f32.slice/crio-9f6ef4d7e567b08256d63a54970994e884b2a60f43461821c196f3c6c1095553 WatchSource:0}: Error finding container 9f6ef4d7e567b08256d63a54970994e884b2a60f43461821c196f3c6c1095553: Status 404 returned error can't find the container with id 9f6ef4d7e567b08256d63a54970994e884b2a60f43461821c196f3c6c1095553 Oct 30 23:33:42.650093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:42.650074 7570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:33:43.203658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:43.203624 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" event={"ID":"bc55b446-6292-4791-a9eb-373860d01f32","Type":"ContainerStarted","Data":"9f6ef4d7e567b08256d63a54970994e884b2a60f43461821c196f3c6c1095553"} Oct 30 23:33:46.222967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:46.222930 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" event={"ID":"bc55b446-6292-4791-a9eb-373860d01f32","Type":"ContainerStarted","Data":"e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b"} Oct 30 23:33:46.240445 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:33:46.240390 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" podStartSLOduration=1.432448557 podStartE2EDuration="4.240378568s" totalImagesPullingTime="2.807930011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:33:46.238796572 +0000 UTC m=+837.757797661" watchObservedRunningTime="2026-10-30 23:33:46.240378568 +0000 UTC m=+837.759379657" Oct 30 23:33:59.411503 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:33:59.411441 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:34:29.411391 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:34:29.411351 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:34:59.411055 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:34:59.410970 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:35:09.021478 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.021429 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv"] Oct 30 23:35:09.024926 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.024906 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.033537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.033511 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 23:35:09.033668 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.033607 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rb9gl\"" Oct 30 23:35:09.049387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.049356 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv"] Oct 30 23:35:09.081900 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.081876 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.082037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.081930 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.082037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.081949 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.082037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.082019 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd5d55c-9530-4b15-af81-8a87166097bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.082206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.082098 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.082206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.082151 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.082206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.082179 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.082311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.082252 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.082311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.082295 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plxln\" (UniqueName: \"kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-kube-api-access-plxln\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.182974 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.182949 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.183086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.182977 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.183086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.182993 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd5d55c-9530-4b15-af81-8a87166097bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.183086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.183013 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.183086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.183044 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.183086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.183069 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.183347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.183095 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.183347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.183121 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plxln\" (UniqueName: \"kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-kube-api-access-plxln\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.183347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.183186 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.183530 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.183442 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.183597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.183565 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.183778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.183744 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.183881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.183780 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.184011 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.183991 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd5d55c-9530-4b15-af81-8a87166097bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.185593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.185575 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.185811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.185793 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.195795 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.195769 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plxln\" (UniqueName: \"kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-kube-api-access-plxln\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.195911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.195836 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g8plv\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.338552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.338516 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:09.649727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.649693 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv"] Oct 30 23:35:09.652487 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:35:09.652434 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd5d55c_9530_4b15_af81_8a87166097bb.slice/crio-50f4723b56bbc1c9fe4090df0857827442d9c2fe9095053c3b63474f19767e35 WatchSource:0}: Error finding container 50f4723b56bbc1c9fe4090df0857827442d9c2fe9095053c3b63474f19767e35: Status 404 returned error can't find the container with id 50f4723b56bbc1c9fe4090df0857827442d9c2fe9095053c3b63474f19767e35 Oct 30 23:35:09.948152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.948073 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pswg7"] Oct 30 23:35:09.948301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:09.948264 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" podUID="bc55b446-6292-4791-a9eb-373860d01f32" containerName="authorino" containerID="cri-o://e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b" gracePeriod=30 Oct 30 23:35:10.200894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.200843 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" Oct 30 23:35:10.294687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.294656 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc55b446-6292-4791-a9eb-373860d01f32-kube-api-access-wxrhs\" (UniqueName: \"kubernetes.io/projected/bc55b446-6292-4791-a9eb-373860d01f32-kube-api-access-wxrhs\") pod \"bc55b446-6292-4791-a9eb-373860d01f32\" (UID: \"bc55b446-6292-4791-a9eb-373860d01f32\") " Oct 30 23:35:10.296977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.296950 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc55b446-6292-4791-a9eb-373860d01f32-kube-api-access-wxrhs" pod "bc55b446-6292-4791-a9eb-373860d01f32" (UID: "bc55b446-6292-4791-a9eb-373860d01f32"). InnerVolumeSpecName "kube-api-access-wxrhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:35:10.395375 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.395340 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxrhs\" (UniqueName: \"kubernetes.io/projected/bc55b446-6292-4791-a9eb-373860d01f32-kube-api-access-wxrhs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:35:10.586967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.586932 7570 generic.go:356] "Generic (PLEG): container finished" podID="bc55b446-6292-4791-a9eb-373860d01f32" containerID="e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b" exitCode=0 Oct 30 23:35:10.587144 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.586984 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" Oct 30 23:35:10.587144 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.587023 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" event={"ID":"bc55b446-6292-4791-a9eb-373860d01f32","Type":"ContainerDied","Data":"e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b"} Oct 30 23:35:10.587144 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.587075 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pswg7" event={"ID":"bc55b446-6292-4791-a9eb-373860d01f32","Type":"ContainerDied","Data":"9f6ef4d7e567b08256d63a54970994e884b2a60f43461821c196f3c6c1095553"} Oct 30 23:35:10.587144 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.587093 7570 scope.go:122] "RemoveContainer" containerID="e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b" Oct 30 23:35:10.588448 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.588373 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" event={"ID":"2dd5d55c-9530-4b15-af81-8a87166097bb","Type":"ContainerStarted","Data":"50f4723b56bbc1c9fe4090df0857827442d9c2fe9095053c3b63474f19767e35"} Oct 30 23:35:10.595988 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.595972 7570 scope.go:122] "RemoveContainer" containerID="e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b" Oct 30 23:35:10.596267 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:35:10.596247 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b\": container with ID starting with e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b not found: ID does not exist" containerID="e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b" Oct 30 23:35:10.596333 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.596278 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b"} err="failed to get container status \"e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b\": rpc error: code = NotFound desc = could not find container \"e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b\": container with ID starting with e78b7cad4b5aaefbf426525f042a8ba8a444bc356e9b617282cffa52b021043b not found: ID does not exist" Oct 30 23:35:10.608615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.608592 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pswg7"] Oct 30 23:35:10.612735 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:10.612714 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pswg7"] Oct 30 23:35:12.602312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:12.602279 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc55b446-6292-4791-a9eb-373860d01f32" path="/var/lib/kubelet/pods/bc55b446-6292-4791-a9eb-373860d01f32/volumes" Oct 30 23:35:13.179826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:13.179733 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:35:13.179936 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:13.179871 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:35:13.179936 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:13.179927 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:35:13.616718 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:13.616682 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" event={"ID":"2dd5d55c-9530-4b15-af81-8a87166097bb","Type":"ContainerStarted","Data":"8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf"} Oct 30 23:35:13.646839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:13.646785 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" podStartSLOduration=1.122423758 podStartE2EDuration="4.646768562s" totalImagesPullingTime="3.524344804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:35:13.645302373 +0000 UTC m=+925.164303464" watchObservedRunningTime="2026-10-30 23:35:13.646768562 +0000 UTC m=+925.165769651" Oct 30 23:35:14.339303 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:14.339255 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:14.340884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:14.340849 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g8plv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 30 23:35:14.341032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:14.340904 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" podUID="2dd5d55c-9530-4b15-af81-8a87166097bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 30 23:35:15.339188 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:15.339142 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g8plv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 30 23:35:15.339711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:15.339244 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" podUID="2dd5d55c-9530-4b15-af81-8a87166097bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 30 23:35:16.339248 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:16.339212 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g8plv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 30 23:35:16.339714 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:16.339288 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" podUID="2dd5d55c-9530-4b15-af81-8a87166097bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 30 23:35:17.339555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:17.339517 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g8plv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 30 23:35:17.340009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:17.339572 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" podUID="2dd5d55c-9530-4b15-af81-8a87166097bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 30 23:35:18.339928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:18.339886 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g8plv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 30 23:35:18.340332 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:18.339943 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" podUID="2dd5d55c-9530-4b15-af81-8a87166097bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 30 23:35:19.343071 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:19.343040 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:19.343540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:19.343388 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:19.344073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:19.344058 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:35:29.410174 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:35:29.410132 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:35:54.082122 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.082090 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9"] Oct 30 23:35:54.082581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.082527 7570 cpu_manager.go:414] "containerMap: removing container" podUID="bc55b446-6292-4791-a9eb-373860d01f32" containerName="authorino" Oct 30 23:35:54.082581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.082540 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc55b446-6292-4791-a9eb-373860d01f32" containerName="authorino" Oct 30 23:35:54.082654 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.082640 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc55b446-6292-4791-a9eb-373860d01f32" containerName="authorino" Oct 30 23:35:54.087081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.087064 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.100596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.100557 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g"] Oct 30 23:35:54.104436 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.104415 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9"] Oct 30 23:35:54.104580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.104565 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.119688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.119664 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g"] Oct 30 23:35:54.172209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172179 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.172342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172215 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.172342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172237 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.172342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172293 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.172479 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172406 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.172479 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172471 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d621d9-9a83-45c7-bef8-9255f543c6f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.172566 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172510 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f99dr\" (UniqueName: \"kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-kube-api-access-f99dr\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.172566 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172532 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.172566 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172548 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zqds\" (UniqueName: \"kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-kube-api-access-2zqds\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.172670 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172622 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.172670 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172641 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f37a547-8183-4386-9f68-d4841e3a252a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.172742 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172692 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f37a547-8183-4386-9f68-d4841e3a252a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.172787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172742 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.172787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172763 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.172857 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172823 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.172857 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172848 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.172921 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172863 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.172921 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.172908 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.273620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.273590 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.273759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.273625 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.273759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.273647 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.273759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.273672 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.273759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.273702 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.273759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.273742 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.274067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.273766 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.274067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.273922 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.274067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.273969 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d621d9-9a83-45c7-bef8-9255f543c6f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.274067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274000 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f99dr\" (UniqueName: \"kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-kube-api-access-f99dr\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.274067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274033 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.274067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274059 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zqds\" (UniqueName: \"kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-kube-api-access-2zqds\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.274361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274077 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.274361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274106 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.274361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274137 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.274361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274168 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f37a547-8183-4386-9f68-d4841e3a252a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.274361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274178 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.274361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274212 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f37a547-8183-4386-9f68-d4841e3a252a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.274361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274265 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.274361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274297 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.274361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274341 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.274842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274450 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.274842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274743 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.274842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274746 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d621d9-9a83-45c7-bef8-9255f543c6f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.274842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274817 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.275000 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.274989 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.275200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.275146 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f37a547-8183-4386-9f68-d4841e3a252a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.275200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.275197 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.276391 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.276365 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.276745 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.276719 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.277142 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.277122 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f37a547-8183-4386-9f68-d4841e3a252a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.277187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.277147 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.286863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.286479 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.286863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.286808 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f99dr\" (UniqueName: \"kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-kube-api-access-f99dr\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.286863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.286821 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zqds\" (UniqueName: \"kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-kube-api-access-2zqds\") pod \"maas-default-gateway-openshift-default-5df48fcc99-t7b8g\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.287706 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.287689 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r5qv9\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.401431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.401348 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:54.419862 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.419840 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:54.690642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.690613 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9"] Oct 30 23:35:54.693244 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:35:54.693215 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f37a547_8183_4386_9f68_d4841e3a252a.slice/crio-4a688d8e7ce9db68252aa18b9cd3a8c21bf8fea077fdd6e4edff4120181d214f WatchSource:0}: Error finding container 4a688d8e7ce9db68252aa18b9cd3a8c21bf8fea077fdd6e4edff4120181d214f: Status 404 returned error can't find the container with id 4a688d8e7ce9db68252aa18b9cd3a8c21bf8fea077fdd6e4edff4120181d214f Oct 30 23:35:54.695198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.695167 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:35:54.695268 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.695228 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:35:54.695268 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.695254 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:35:54.711120 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.711035 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g"] Oct 30 23:35:54.719207 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:35:54.719185 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d621d9_9a83_45c7_bef8_9255f543c6f0.slice/crio-d5212bf09fe33ededa07b37c5fd4c6875d2514a050be0030a4f0a7730fe5a5fb WatchSource:0}: Error finding container d5212bf09fe33ededa07b37c5fd4c6875d2514a050be0030a4f0a7730fe5a5fb: Status 404 returned error can't find the container with id d5212bf09fe33ededa07b37c5fd4c6875d2514a050be0030a4f0a7730fe5a5fb Oct 30 23:35:54.721369 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.721342 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:35:54.721439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.721399 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:35:54.721439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.721427 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:35:54.799143 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.799096 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" event={"ID":"8f37a547-8183-4386-9f68-d4841e3a252a","Type":"ContainerStarted","Data":"b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035"} Oct 30 23:35:54.799264 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.799168 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" event={"ID":"8f37a547-8183-4386-9f68-d4841e3a252a","Type":"ContainerStarted","Data":"4a688d8e7ce9db68252aa18b9cd3a8c21bf8fea077fdd6e4edff4120181d214f"} Oct 30 23:35:54.801643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.801617 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" event={"ID":"02d621d9-9a83-45c7-bef8-9255f543c6f0","Type":"ContainerStarted","Data":"9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855"} Oct 30 23:35:54.801750 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.801667 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" event={"ID":"02d621d9-9a83-45c7-bef8-9255f543c6f0","Type":"ContainerStarted","Data":"d5212bf09fe33ededa07b37c5fd4c6875d2514a050be0030a4f0a7730fe5a5fb"} Oct 30 23:35:54.822337 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:54.822019 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" podStartSLOduration=0.822002328 podStartE2EDuration="822.002328ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:35:54.821636467 +0000 UTC m=+966.340637557" watchObservedRunningTime="2026-10-30 23:35:54.822002328 +0000 UTC m=+966.341003419" Oct 30 23:35:55.403296 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:55.403251 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:35:55.404019 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:55.403995 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r5qv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 23:35:55.404194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:55.404166 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" podUID="8f37a547-8183-4386-9f68-d4841e3a252a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 23:35:55.421909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:55.421833 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:35:55.424049 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:55.423741 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-t7b8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 23:35:55.424049 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:55.423799 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 23:35:56.402928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:56.402887 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r5qv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 23:35:56.403233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:56.403206 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" podUID="8f37a547-8183-4386-9f68-d4841e3a252a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 23:35:56.421904 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:56.421856 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-t7b8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 23:35:56.422498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:56.422443 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 23:35:57.407939 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:57.407898 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r5qv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 23:35:57.408237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:57.408203 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" podUID="8f37a547-8183-4386-9f68-d4841e3a252a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 23:35:57.422950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:57.422836 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-t7b8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 23:35:57.422950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:57.422895 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 23:35:58.402187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:58.401904 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r5qv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 23:35:58.402187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:58.401968 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" podUID="8f37a547-8183-4386-9f68-d4841e3a252a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 23:35:58.441531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:58.436118 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-t7b8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 23:35:58.441531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:58.436203 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 23:35:59.408584 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:59.408075 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r5qv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 23:35:59.408584 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:59.408151 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" podUID="8f37a547-8183-4386-9f68-d4841e3a252a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 23:35:59.422950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:59.422804 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-t7b8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 23:35:59.422950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:35:59.422879 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 23:35:59.458587 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:35:59.458549 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:36:00.403875 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:00.403833 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r5qv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 23:36:00.404168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:00.404133 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" podUID="8f37a547-8183-4386-9f68-d4841e3a252a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 23:36:00.427979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:00.427940 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-t7b8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 23:36:00.428227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:00.428200 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 23:36:01.406337 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:01.406304 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:36:01.406808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:01.406587 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:36:01.407416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:01.407397 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:36:01.423920 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:01.423900 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:36:01.424207 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:01.424179 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:36:01.424812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:01.424798 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:36:01.432696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:01.432652 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podStartSLOduration=7.432638849 podStartE2EDuration="7.432638849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:35:54.846024163 +0000 UTC m=+966.365025257" watchObservedRunningTime="2026-10-30 23:36:01.432638849 +0000 UTC m=+972.951639935" Oct 30 23:36:02.030582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.030550 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs"] Oct 30 23:36:02.034442 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.034423 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.039874 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.039855 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v5mxr\"" Oct 30 23:36:02.042829 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.042811 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 23:36:02.042829 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.042812 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 23:36:02.047701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.047681 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs"] Oct 30 23:36:02.093010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.092981 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f87cf2ab-c80c-4680-9c10-497e2a11235b-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.093196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.093027 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f87cf2ab-c80c-4680-9c10-497e2a11235b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.093196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.093065 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f87cf2ab-c80c-4680-9c10-497e2a11235b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.093196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.093145 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f87cf2ab-c80c-4680-9c10-497e2a11235b-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.093307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.093228 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f87cf2ab-c80c-4680-9c10-497e2a11235b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.093307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.093289 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7968\" (UniqueName: \"kubernetes.io/projected/f87cf2ab-c80c-4680-9c10-497e2a11235b-kube-api-access-r7968\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.142613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.142586 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 23:36:02.193998 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.193969 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7968\" (UniqueName: \"kubernetes.io/projected/f87cf2ab-c80c-4680-9c10-497e2a11235b-kube-api-access-r7968\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.194187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.194027 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f87cf2ab-c80c-4680-9c10-497e2a11235b-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.194187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.194058 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f87cf2ab-c80c-4680-9c10-497e2a11235b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.194187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.194082 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f87cf2ab-c80c-4680-9c10-497e2a11235b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.194187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.194098 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f87cf2ab-c80c-4680-9c10-497e2a11235b-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.194414 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.194227 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f87cf2ab-c80c-4680-9c10-497e2a11235b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.194557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.194531 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f87cf2ab-c80c-4680-9c10-497e2a11235b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.194557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.194549 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f87cf2ab-c80c-4680-9c10-497e2a11235b-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.194678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.194630 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f87cf2ab-c80c-4680-9c10-497e2a11235b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.196694 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.196671 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f87cf2ab-c80c-4680-9c10-497e2a11235b-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.197072 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.197041 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f87cf2ab-c80c-4680-9c10-497e2a11235b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.206582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.206562 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7968\" (UniqueName: \"kubernetes.io/projected/f87cf2ab-c80c-4680-9c10-497e2a11235b-kube-api-access-r7968\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6pxqs\" (UID: \"f87cf2ab-c80c-4680-9c10-497e2a11235b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.347632 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.347602 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:02.630860 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.630728 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs"] Oct 30 23:36:02.868350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:02.868297 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" event={"ID":"f87cf2ab-c80c-4680-9c10-497e2a11235b","Type":"ContainerStarted","Data":"163dac5b00b41919b217b85bb2813b3915be499d337d50c62b5db28fcbcc697c"} Oct 30 23:36:03.079358 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.079327 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx"] Oct 30 23:36:03.083499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.083456 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.095167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.094492 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx"] Oct 30 23:36:03.095906 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.095640 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 23:36:03.211535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.211493 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.211707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.211553 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxw7\" (UniqueName: \"kubernetes.io/projected/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-kube-api-access-wgxw7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.211707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.211693 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.211805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.211777 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.211838 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.211811 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.211870 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.211833 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.313353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.313317 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.313538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.313382 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.313538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.313404 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.313538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.313422 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.313538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.313437 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.313538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.313455 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxw7\" (UniqueName: \"kubernetes.io/projected/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-kube-api-access-wgxw7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.313896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.313869 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.313982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.313918 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.313982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.313945 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.316066 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.316041 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.316323 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.316302 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.326126 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.326105 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxw7\" (UniqueName: \"kubernetes.io/projected/dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e-kube-api-access-wgxw7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx\" (UID: \"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.401637 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.401562 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:03.721554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.721383 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx"] Oct 30 23:36:03.724908 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:36:03.724877 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6cf0ef_7ff3_4c02_8e20_bc39ac49bd9e.slice/crio-719b94b48eb408dceebe53cb63e54dd2baa988467f24c04a53324d812dd1371c WatchSource:0}: Error finding container 719b94b48eb408dceebe53cb63e54dd2baa988467f24c04a53324d812dd1371c: Status 404 returned error can't find the container with id 719b94b48eb408dceebe53cb63e54dd2baa988467f24c04a53324d812dd1371c Oct 30 23:36:03.876756 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:03.876714 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" event={"ID":"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e","Type":"ContainerStarted","Data":"719b94b48eb408dceebe53cb63e54dd2baa988467f24c04a53324d812dd1371c"} Oct 30 23:36:04.885895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:04.885859 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" event={"ID":"f87cf2ab-c80c-4680-9c10-497e2a11235b","Type":"ContainerStarted","Data":"066dae52ca122c4497dfb9780665148398a7cb4c2052036730d3abef2765dff7"} Oct 30 23:36:04.913294 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:04.913245 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podStartSLOduration=1.07407732 podStartE2EDuration="2.913231936s" totalImagesPullingTime="1.839154616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:36:04.912839367 +0000 UTC m=+976.431840469" watchObservedRunningTime="2026-10-30 23:36:04.913231936 +0000 UTC m=+976.432233025" Oct 30 23:36:05.347753 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:05.347720 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:05.891936 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:05.891888 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" event={"ID":"dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e","Type":"ContainerStarted","Data":"6a107c7d21fe06a2ff794b8a280e507613f51b503b3ee5431a319e82fce7c79e"} Oct 30 23:36:05.913744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:05.913696 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podStartSLOduration=1.759724251 podStartE2EDuration="2.913682025s" totalImagesPullingTime="1.153957774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:36:05.911991126 +0000 UTC m=+977.430992217" watchObservedRunningTime="2026-10-30 23:36:05.913682025 +0000 UTC m=+977.432683114" Oct 30 23:36:06.402288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.402248 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:06.509521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.509480 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2"] Oct 30 23:36:06.514044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.514019 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.521627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.521597 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2"] Oct 30 23:36:06.526501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.525692 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 23:36:06.665065 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.664963 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nxkq\" (UniqueName: \"kubernetes.io/projected/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-kube-api-access-9nxkq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.665065 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.665016 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.665284 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.665202 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.665284 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.665261 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.665392 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.665368 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.665506 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.665449 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.766334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.766276 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.766334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.766335 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.766623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.766582 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nxkq\" (UniqueName: \"kubernetes.io/projected/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-kube-api-access-9nxkq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.766689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.766635 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.766748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.766720 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.766810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.766751 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.766810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.766761 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.766810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.766786 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.767208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.767179 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.770074 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.770045 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.770367 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.770345 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.779845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.779818 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nxkq\" (UniqueName: \"kubernetes.io/projected/5275c1e3-85ea-449f-b526-7c3cbb14ed4c-kube-api-access-9nxkq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2\" (UID: \"5275c1e3-85ea-449f-b526-7c3cbb14ed4c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:06.839964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:06.839674 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:07.262566 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:07.260990 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2"] Oct 30 23:36:07.265016 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:36:07.264985 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5275c1e3_85ea_449f_b526_7c3cbb14ed4c.slice/crio-090b465b62f88735e5410e490daaccc5e82e93eb8937cfbe314b593992bab4f6 WatchSource:0}: Error finding container 090b465b62f88735e5410e490daaccc5e82e93eb8937cfbe314b593992bab4f6: Status 404 returned error can't find the container with id 090b465b62f88735e5410e490daaccc5e82e93eb8937cfbe314b593992bab4f6 Oct 30 23:36:07.912307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:07.912199 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" event={"ID":"5275c1e3-85ea-449f-b526-7c3cbb14ed4c","Type":"ContainerStarted","Data":"3ca18ce1dcb30eb331f4d47b1e7e5be54c5b4dc4a85f8ebd290801a6e7523b36"} Oct 30 23:36:07.912307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:07.912256 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" event={"ID":"5275c1e3-85ea-449f-b526-7c3cbb14ed4c","Type":"ContainerStarted","Data":"090b465b62f88735e5410e490daaccc5e82e93eb8937cfbe314b593992bab4f6"} Oct 30 23:36:07.934786 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:07.933484 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podStartSLOduration=1.567153118 podStartE2EDuration="1.933451829s" totalImagesPullingTime="366.298711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:36:07.932965492 +0000 UTC m=+979.451966584" watchObservedRunningTime="2026-10-30 23:36:07.933451829 +0000 UTC m=+979.452452920" Oct 30 23:36:08.840413 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:08.840378 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:11.389308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.389274 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888"] Oct 30 23:36:11.393846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.393823 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.398713 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.398693 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 23:36:11.404321 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.404296 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888"] Oct 30 23:36:11.526696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.526662 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeffad66-ea6b-486f-abd8-63e68120af4c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.526877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.526709 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg6nx\" (UniqueName: \"kubernetes.io/projected/eeffad66-ea6b-486f-abd8-63e68120af4c-kube-api-access-jg6nx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.526877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.526825 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeffad66-ea6b-486f-abd8-63e68120af4c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.527037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.526884 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeffad66-ea6b-486f-abd8-63e68120af4c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.527037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.526914 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeffad66-ea6b-486f-abd8-63e68120af4c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.527116 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.527069 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeffad66-ea6b-486f-abd8-63e68120af4c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.627961 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.627928 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeffad66-ea6b-486f-abd8-63e68120af4c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.627961 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.627966 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg6nx\" (UniqueName: \"kubernetes.io/projected/eeffad66-ea6b-486f-abd8-63e68120af4c-kube-api-access-jg6nx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.628206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.628003 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeffad66-ea6b-486f-abd8-63e68120af4c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.628206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.628056 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeffad66-ea6b-486f-abd8-63e68120af4c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.628206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.628089 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeffad66-ea6b-486f-abd8-63e68120af4c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.628206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.628138 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeffad66-ea6b-486f-abd8-63e68120af4c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.628597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.628560 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeffad66-ea6b-486f-abd8-63e68120af4c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.628711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.628604 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeffad66-ea6b-486f-abd8-63e68120af4c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.628711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.628662 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeffad66-ea6b-486f-abd8-63e68120af4c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.630493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.630474 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeffad66-ea6b-486f-abd8-63e68120af4c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.630685 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.630667 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeffad66-ea6b-486f-abd8-63e68120af4c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.640778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.640715 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg6nx\" (UniqueName: \"kubernetes.io/projected/eeffad66-ea6b-486f-abd8-63e68120af4c-kube-api-access-jg6nx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-c6888\" (UID: \"eeffad66-ea6b-486f-abd8-63e68120af4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.709396 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.709360 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:11.988362 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:11.988336 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888"] Oct 30 23:36:11.990782 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:36:11.990752 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeffad66_ea6b_486f_abd8_63e68120af4c.slice/crio-9c6ddb3b5e984c84504d4c378a3657435cfe5440d840e07afdd9113f74dd9d5c WatchSource:0}: Error finding container 9c6ddb3b5e984c84504d4c378a3657435cfe5440d840e07afdd9113f74dd9d5c: Status 404 returned error can't find the container with id 9c6ddb3b5e984c84504d4c378a3657435cfe5440d840e07afdd9113f74dd9d5c Oct 30 23:36:12.348789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:12.348767 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:12.361545 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:12.361521 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:12.950267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:12.950224 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" event={"ID":"eeffad66-ea6b-486f-abd8-63e68120af4c","Type":"ContainerStarted","Data":"ff182574a2799ed15ce1d29464abefba4fe8ce1908e768b5abe9d22c9296a9f1"} Oct 30 23:36:12.950744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:12.950294 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" event={"ID":"eeffad66-ea6b-486f-abd8-63e68120af4c","Type":"ContainerStarted","Data":"9c6ddb3b5e984c84504d4c378a3657435cfe5440d840e07afdd9113f74dd9d5c"} Oct 30 23:36:12.962056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:12.962034 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:36:12.972594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:12.972549 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podStartSLOduration=1.657314967 podStartE2EDuration="1.972535963s" totalImagesPullingTime="315.220996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:36:12.971406705 +0000 UTC m=+984.490407808" watchObservedRunningTime="2026-10-30 23:36:12.972535963 +0000 UTC m=+984.491537053" Oct 30 23:36:13.401929 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:13.401889 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:13.421486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:13.421434 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:13.710271 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:13.710168 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:13.974518 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:13.974409 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:36:16.840385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:16.840344 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:16.853027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:16.853005 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:16.987418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:16.987391 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:36:21.709931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:21.709893 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:21.722441 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:21.722408 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:22.098369 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:22.098344 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:36:23.589283 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.589248 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74dc7574c6-6rws8"] Oct 30 23:36:23.593285 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.593263 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74dc7574c6-6rws8" Oct 30 23:36:23.600913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.600883 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74dc7574c6-6rws8"] Oct 30 23:36:23.602834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.602812 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 23:36:23.603936 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.603914 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 23:36:23.604004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.603958 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 23:36:23.604004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.603988 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hc9wr\"" Oct 30 23:36:23.611867 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.611844 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8mk4\" (UniqueName: \"kubernetes.io/projected/3dc85af3-2ef5-4005-a8c5-d515a92d2eee-kube-api-access-d8mk4\") pod \"authorino-74dc7574c6-6rws8\" (UID: \"3dc85af3-2ef5-4005-a8c5-d515a92d2eee\") " pod="kuadrant-system/authorino-74dc7574c6-6rws8" Oct 30 23:36:23.611981 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.611899 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3dc85af3-2ef5-4005-a8c5-d515a92d2eee-tls-cert\") pod \"authorino-74dc7574c6-6rws8\" (UID: \"3dc85af3-2ef5-4005-a8c5-d515a92d2eee\") " pod="kuadrant-system/authorino-74dc7574c6-6rws8" Oct 30 23:36:23.712680 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.712647 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3dc85af3-2ef5-4005-a8c5-d515a92d2eee-tls-cert\") pod \"authorino-74dc7574c6-6rws8\" (UID: \"3dc85af3-2ef5-4005-a8c5-d515a92d2eee\") " pod="kuadrant-system/authorino-74dc7574c6-6rws8" Oct 30 23:36:23.712884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.712800 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8mk4\" (UniqueName: \"kubernetes.io/projected/3dc85af3-2ef5-4005-a8c5-d515a92d2eee-kube-api-access-d8mk4\") pod \"authorino-74dc7574c6-6rws8\" (UID: \"3dc85af3-2ef5-4005-a8c5-d515a92d2eee\") " pod="kuadrant-system/authorino-74dc7574c6-6rws8" Oct 30 23:36:23.715207 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.715184 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3dc85af3-2ef5-4005-a8c5-d515a92d2eee-tls-cert\") pod \"authorino-74dc7574c6-6rws8\" (UID: \"3dc85af3-2ef5-4005-a8c5-d515a92d2eee\") " pod="kuadrant-system/authorino-74dc7574c6-6rws8" Oct 30 23:36:23.725394 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.725371 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8mk4\" (UniqueName: \"kubernetes.io/projected/3dc85af3-2ef5-4005-a8c5-d515a92d2eee-kube-api-access-d8mk4\") pod \"authorino-74dc7574c6-6rws8\" (UID: \"3dc85af3-2ef5-4005-a8c5-d515a92d2eee\") " pod="kuadrant-system/authorino-74dc7574c6-6rws8" Oct 30 23:36:23.908307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:23.908224 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74dc7574c6-6rws8" Oct 30 23:36:24.148173 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.148144 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c"] Oct 30 23:36:24.152984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.152967 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.168205 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.168177 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c"] Oct 30 23:36:24.192286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.192260 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74dc7574c6-6rws8"] Oct 30 23:36:24.195496 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:36:24.195421 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dc85af3_2ef5_4005_a8c5_d515a92d2eee.slice/crio-800e321e2e3961fca290afc839d5751c184bb2c4919ec88dc62c122cc35898b7 WatchSource:0}: Error finding container 800e321e2e3961fca290afc839d5751c184bb2c4919ec88dc62c122cc35898b7: Status 404 returned error can't find the container with id 800e321e2e3961fca290afc839d5751c184bb2c4919ec88dc62c122cc35898b7 Oct 30 23:36:24.218500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.218457 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.218621 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.218595 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.218695 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.218633 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.218695 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.218665 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.218782 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.218728 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w58l\" (UniqueName: \"kubernetes.io/projected/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-kube-api-access-6w58l\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.218782 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.218753 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.218886 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.218830 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.218886 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.218861 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.218970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.218906 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.319473 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.319438 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.319620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.319499 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.319620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.319524 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.319620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.319549 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w58l\" (UniqueName: \"kubernetes.io/projected/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-kube-api-access-6w58l\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.319620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.319565 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.319620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.319603 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.319888 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.319627 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.319888 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.319670 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.319888 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.319816 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.320067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.319895 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.320067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.319935 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.320067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.319971 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.320226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.320120 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.320454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.320435 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.322058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.322037 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.322256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.322240 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.332398 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.332375 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.332535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.332515 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w58l\" (UniqueName: \"kubernetes.io/projected/e7406dbc-b7ad-4b0c-ac37-c85b302bf76e-kube-api-access-6w58l\") pod \"maas-default-gateway-openshift-default-5df48fcc99-7rp6c\" (UID: \"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.467225 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.467149 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:24.757143 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.757003 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c"] Oct 30 23:36:24.759419 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:36:24.759389 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7406dbc_b7ad_4b0c_ac37_c85b302bf76e.slice/crio-fc640e18336435686f7e3fb9a66b2bdb7ad06357eda4f3ecb011ac5ab3ea8786 WatchSource:0}: Error finding container fc640e18336435686f7e3fb9a66b2bdb7ad06357eda4f3ecb011ac5ab3ea8786: Status 404 returned error can't find the container with id fc640e18336435686f7e3fb9a66b2bdb7ad06357eda4f3ecb011ac5ab3ea8786 Oct 30 23:36:24.761844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.761809 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:36:24.761972 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.761893 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:36:24.761972 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:24.761939 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:36:25.102510 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:25.102450 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74dc7574c6-6rws8" event={"ID":"3dc85af3-2ef5-4005-a8c5-d515a92d2eee","Type":"ContainerStarted","Data":"800e321e2e3961fca290afc839d5751c184bb2c4919ec88dc62c122cc35898b7"} Oct 30 23:36:25.104121 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:25.104096 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" event={"ID":"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e","Type":"ContainerStarted","Data":"110b0ab2870e851893198c3c9105401cbce22fb9931cd1453dd4976f0d3d0369"} Oct 30 23:36:25.104292 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:25.104143 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" event={"ID":"e7406dbc-b7ad-4b0c-ac37-c85b302bf76e","Type":"ContainerStarted","Data":"fc640e18336435686f7e3fb9a66b2bdb7ad06357eda4f3ecb011ac5ab3ea8786"} Oct 30 23:36:25.128781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:25.128730 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podStartSLOduration=1.128712041 podStartE2EDuration="1.128712041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:36:25.128001163 +0000 UTC m=+996.647002279" watchObservedRunningTime="2026-10-30 23:36:25.128712041 +0000 UTC m=+996.647713132" Oct 30 23:36:25.467342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:25.467311 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:25.468790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:25.468764 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 23:36:25.468887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:25.468815 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 23:36:26.120819 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:26.120594 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74dc7574c6-6rws8" event={"ID":"3dc85af3-2ef5-4005-a8c5-d515a92d2eee","Type":"ContainerStarted","Data":"687204482ade1a753a6900a662613e39a24edde7958e1e93edda91cc4cac255a"} Oct 30 23:36:26.152063 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:26.152009 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-74dc7574c6-6rws8" podStartSLOduration=1.917948901 podStartE2EDuration="3.151993919s" totalImagesPullingTime="1.234045018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:36:26.151150838 +0000 UTC m=+997.670151927" watchObservedRunningTime="2026-10-30 23:36:26.151993919 +0000 UTC m=+997.670995009" Oct 30 23:36:26.468077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:26.467978 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 23:36:26.468077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:26.468042 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 23:36:27.468293 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:27.468253 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 23:36:27.468726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:27.468309 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 23:36:28.467970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:28.467935 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 23:36:28.468154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:28.467993 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 23:36:29.437951 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:36:29.437904 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:36:29.468527 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:29.468497 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 23:36:29.468694 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:29.468565 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 23:36:30.468007 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:30.467965 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 23:36:30.468403 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:30.468034 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 23:36:31.468319 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:31.468275 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 23:36:31.468741 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:31.468360 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 23:36:33.294576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:33.294536 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:33.295097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:33.294988 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:33.295927 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:36:33.295909 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:36:59.412004 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:36:59.411957 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:37:25.475549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.475518 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7"] Oct 30 23:37:25.479551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.479526 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.487813 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.487771 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dd3a15-openshift-default-dockercfg-72cjz\"" Oct 30 23:37:25.494074 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.494019 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.494369 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.494348 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.494484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.494411 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.494484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.494443 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.494605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.494499 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31585445-5877-4947-8df5-f30317948127-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.494605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.494532 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.494704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.494624 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.494704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.494664 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31585445-5877-4947-8df5-f30317948127-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.494805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.494709 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fssz2\" (UniqueName: \"kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-kube-api-access-fssz2\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.495983 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.495957 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7"] Oct 30 23:37:25.570445 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.570411 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4"] Oct 30 23:37:25.574900 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.574878 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.579885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.579864 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-611ef3-openshift-default-dockercfg-gcsfx\"" Oct 30 23:37:25.595854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.595816 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.596035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.595868 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fssz2\" (UniqueName: \"kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-kube-api-access-fssz2\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.596035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.595905 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnbjz\" (UniqueName: \"kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-kube-api-access-nnbjz\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.596035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.595928 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.596035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.595961 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.596035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.595974 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4"] Oct 30 23:37:25.596035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.595988 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.596035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.596021 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.596370 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.596314 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.596430 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.596411 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.596520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.596453 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.596587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.596550 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.596632 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.596582 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.596693 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.596639 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31585445-5877-4947-8df5-f30317948127-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.596742 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.596702 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.596799 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.596772 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.596866 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.596849 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.596922 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.596883 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.596978 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.596922 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31585445-5877-4947-8df5-f30317948127-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.597205 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.597177 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.597785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.597418 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.597785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.597742 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.598061 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.598034 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31585445-5877-4947-8df5-f30317948127-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.598337 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.598317 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.598923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.598901 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.600178 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.600152 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31585445-5877-4947-8df5-f30317948127-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.615545 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.615521 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.620013 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.619986 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fssz2\" (UniqueName: \"kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-kube-api-access-fssz2\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.698036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.697996 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.698215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.698053 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnbjz\" (UniqueName: \"kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-kube-api-access-nnbjz\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.698215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.698082 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.698380 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.698233 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.698380 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.698276 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.698380 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.698364 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.698602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.698394 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.698602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.698453 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.698602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.698528 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.698766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.698605 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.698766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.698667 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.699069 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.699044 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.699236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.699217 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.699360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.699343 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.701851 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.701824 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.701996 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.701978 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.715418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.715394 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnbjz\" (UniqueName: \"kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-kube-api-access-nnbjz\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.716897 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.716717 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.793513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.793434 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:25.888375 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.888342 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:25.905509 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.904842 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl"] Oct 30 23:37:25.908973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.908947 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:25.924596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.924566 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b567bb-openshift-default-dockercfg-wvjgm\"" Oct 30 23:37:25.928805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:25.928235 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl"] Oct 30 23:37:26.004968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.004921 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f473a6-9517-4367-b935-294dbd28ceec-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.005175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.004998 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.005175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.005140 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.005316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.005196 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.005316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.005287 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplkm\" (UniqueName: \"kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-kube-api-access-gplkm\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.005425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.005346 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.005425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.005394 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.005574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.005421 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.005574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.005485 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f473a6-9517-4367-b935-294dbd28ceec-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.075976 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.075950 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7"] Oct 30 23:37:26.078749 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:37:26.078727 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31585445_5877_4947_8df5_f30317948127.slice/crio-adcc5478966b2e686e68e2158a5ffab2f14af801021746655d1fb2477c487796 WatchSource:0}: Error finding container adcc5478966b2e686e68e2158a5ffab2f14af801021746655d1fb2477c487796: Status 404 returned error can't find the container with id adcc5478966b2e686e68e2158a5ffab2f14af801021746655d1fb2477c487796 Oct 30 23:37:26.081061 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.081029 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:26.081124 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.081093 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:26.081159 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.081120 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:26.106974 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.106941 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplkm\" (UniqueName: \"kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-kube-api-access-gplkm\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.107123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.107072 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.107641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.107236 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.107641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.107277 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.107641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.107312 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f473a6-9517-4367-b935-294dbd28ceec-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.107641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.107355 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f473a6-9517-4367-b935-294dbd28ceec-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.107641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.107405 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.107641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.107501 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.107641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.107566 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.108059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.107826 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.108059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.107896 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.108165 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.108065 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.108165 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.108093 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f473a6-9517-4367-b935-294dbd28ceec-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.108319 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.108302 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.109622 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.109606 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.110635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.110618 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f473a6-9517-4367-b935-294dbd28ceec-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.122631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.122611 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.122717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.122658 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplkm\" (UniqueName: \"kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-kube-api-access-gplkm\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.181564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.181474 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4"] Oct 30 23:37:26.184479 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:37:26.184429 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fab750_eae1_4f10_b8f3_3db73b00a1f9.slice/crio-f81d6eccbe6b875d8bfed0431580df91c7029cbb984e47ce4be672f8c2c2f1f6 WatchSource:0}: Error finding container f81d6eccbe6b875d8bfed0431580df91c7029cbb984e47ce4be672f8c2c2f1f6: Status 404 returned error can't find the container with id f81d6eccbe6b875d8bfed0431580df91c7029cbb984e47ce4be672f8c2c2f1f6 Oct 30 23:37:26.187080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.187047 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:26.187157 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.187112 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:26.187157 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.187140 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:26.240000 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.239118 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:26.441649 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.441593 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" event={"ID":"e8fab750-eae1-4f10-b8f3-3db73b00a1f9","Type":"ContainerStarted","Data":"dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca"} Oct 30 23:37:26.441846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.441787 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" event={"ID":"e8fab750-eae1-4f10-b8f3-3db73b00a1f9","Type":"ContainerStarted","Data":"f81d6eccbe6b875d8bfed0431580df91c7029cbb984e47ce4be672f8c2c2f1f6"} Oct 30 23:37:26.443601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.443576 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" event={"ID":"31585445-5877-4947-8df5-f30317948127","Type":"ContainerStarted","Data":"74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261"} Oct 30 23:37:26.443679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.443624 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" event={"ID":"31585445-5877-4947-8df5-f30317948127","Type":"ContainerStarted","Data":"adcc5478966b2e686e68e2158a5ffab2f14af801021746655d1fb2477c487796"} Oct 30 23:37:26.468602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.468540 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podStartSLOduration=1.4685223490000001 podStartE2EDuration="1.468522349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:26.467307315 +0000 UTC m=+1057.986308406" watchObservedRunningTime="2026-10-30 23:37:26.468522349 +0000 UTC m=+1057.987523441" Oct 30 23:37:26.499203 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.499133 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podStartSLOduration=1.499115393 podStartE2EDuration="1.499115393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:26.497053818 +0000 UTC m=+1058.016054909" watchObservedRunningTime="2026-10-30 23:37:26.499115393 +0000 UTC m=+1058.018116484" Oct 30 23:37:26.566874 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.566836 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl"] Oct 30 23:37:26.569068 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:37:26.569039 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f473a6_9517_4367_b935_294dbd28ceec.slice/crio-fbd4314b7b3d8c3cc09a22d3a942278f369347bbfa7a0333bac954721569d92b WatchSource:0}: Error finding container fbd4314b7b3d8c3cc09a22d3a942278f369347bbfa7a0333bac954721569d92b: Status 404 returned error can't find the container with id fbd4314b7b3d8c3cc09a22d3a942278f369347bbfa7a0333bac954721569d92b Oct 30 23:37:26.571704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.571665 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:26.571791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.571739 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:26.571791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.571769 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:26.794600 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.794566 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:26.889257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:26.889165 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:27.450202 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:27.450160 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" event={"ID":"a6f473a6-9517-4367-b935-294dbd28ceec","Type":"ContainerStarted","Data":"8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50"} Oct 30 23:37:27.450389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:27.450219 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" event={"ID":"a6f473a6-9517-4367-b935-294dbd28ceec","Type":"ContainerStarted","Data":"fbd4314b7b3d8c3cc09a22d3a942278f369347bbfa7a0333bac954721569d92b"} Oct 30 23:37:27.480571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:27.480521 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podStartSLOduration=2.480504225 podStartE2EDuration="2.480504225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:27.478934841 +0000 UTC m=+1058.997935933" watchObservedRunningTime="2026-10-30 23:37:27.480504225 +0000 UTC m=+1058.999505320" Oct 30 23:37:27.800275 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:27.800167 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:27.894536 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:27.894505 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:28.240381 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:28.240349 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:28.245793 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:28.245771 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:28.455112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:28.455074 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:28.455658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:28.455146 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:28.455658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:28.455172 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:28.455956 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:28.455937 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:28.456040 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:28.455995 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:28.456040 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:28.456027 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:29.411613 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:37:29.411576 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:37:31.364279 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.364246 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt"] Oct 30 23:37:31.369041 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.369019 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:37:31.379728 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.379705 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-611ef3-dockercfg-m9bvp\"" Oct 30 23:37:31.380598 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.380570 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-611ef3\"" Oct 30 23:37:31.384418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.384392 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt"] Oct 30 23:37:31.488638 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.488601 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhlvw\" (UniqueName: \"kubernetes.io/projected/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-kube-api-access-rhlvw\") pod \"payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt\" (UID: \"7371f1ff-163e-49b2-a4ea-5a9bf3a18215\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:37:31.488801 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.488747 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt\" (UID: \"7371f1ff-163e-49b2-a4ea-5a9bf3a18215\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:37:31.590072 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.590034 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt\" (UID: \"7371f1ff-163e-49b2-a4ea-5a9bf3a18215\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:37:31.590443 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.590414 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhlvw\" (UniqueName: \"kubernetes.io/projected/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-kube-api-access-rhlvw\") pod \"payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt\" (UID: \"7371f1ff-163e-49b2-a4ea-5a9bf3a18215\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:37:31.591069 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.591037 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt\" (UID: \"7371f1ff-163e-49b2-a4ea-5a9bf3a18215\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:37:31.603214 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.603185 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhlvw\" (UniqueName: \"kubernetes.io/projected/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-kube-api-access-rhlvw\") pod \"payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt\" (UID: \"7371f1ff-163e-49b2-a4ea-5a9bf3a18215\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:37:31.685331 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.685248 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:37:31.970956 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:31.970860 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt"] Oct 30 23:37:31.980704 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:37:31.980596 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7371f1ff_163e_49b2_a4ea_5a9bf3a18215.slice/crio-caef89725f86ed7c0ca0ee95652a0fcf1d96ebb1277189ff6746b7d1489cd461 WatchSource:0}: Error finding container caef89725f86ed7c0ca0ee95652a0fcf1d96ebb1277189ff6746b7d1489cd461: Status 404 returned error can't find the container with id caef89725f86ed7c0ca0ee95652a0fcf1d96ebb1277189ff6746b7d1489cd461 Oct 30 23:37:32.035491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.035421 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp"] Oct 30 23:37:32.040174 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.040153 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:37:32.047603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.047576 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-e15304-dockercfg-8mxkq\"" Oct 30 23:37:32.050768 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.048646 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-e15304\"" Oct 30 23:37:32.050768 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.049124 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp"] Oct 30 23:37:32.096313 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.096284 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522aa71b-47e4-44b1-825c-bf81305d21b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp\" (UID: \"522aa71b-47e4-44b1-825c-bf81305d21b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:37:32.096517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.096342 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzq2\" (UniqueName: \"kubernetes.io/projected/522aa71b-47e4-44b1-825c-bf81305d21b4-kube-api-access-pvzq2\") pod \"payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp\" (UID: \"522aa71b-47e4-44b1-825c-bf81305d21b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:37:32.197374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.197338 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522aa71b-47e4-44b1-825c-bf81305d21b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp\" (UID: \"522aa71b-47e4-44b1-825c-bf81305d21b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:37:32.197595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.197406 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzq2\" (UniqueName: \"kubernetes.io/projected/522aa71b-47e4-44b1-825c-bf81305d21b4-kube-api-access-pvzq2\") pod \"payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp\" (UID: \"522aa71b-47e4-44b1-825c-bf81305d21b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:37:32.198073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.198053 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522aa71b-47e4-44b1-825c-bf81305d21b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp\" (UID: \"522aa71b-47e4-44b1-825c-bf81305d21b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:37:32.212269 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.212235 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzq2\" (UniqueName: \"kubernetes.io/projected/522aa71b-47e4-44b1-825c-bf81305d21b4-kube-api-access-pvzq2\") pod \"payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp\" (UID: \"522aa71b-47e4-44b1-825c-bf81305d21b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:37:32.379198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.379163 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:37:32.480678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.480637 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" event={"ID":"7371f1ff-163e-49b2-a4ea-5a9bf3a18215","Type":"ContainerStarted","Data":"caef89725f86ed7c0ca0ee95652a0fcf1d96ebb1277189ff6746b7d1489cd461"} Oct 30 23:37:32.659543 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.659509 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp"] Oct 30 23:37:32.662359 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:37:32.662327 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522aa71b_47e4_44b1_825c_bf81305d21b4.slice/crio-77a6206811d38439424d8cfd4ae9850942b50cf9cc4e90a5ff525bd300babfd2 WatchSource:0}: Error finding container 77a6206811d38439424d8cfd4ae9850942b50cf9cc4e90a5ff525bd300babfd2: Status 404 returned error can't find the container with id 77a6206811d38439424d8cfd4ae9850942b50cf9cc4e90a5ff525bd300babfd2 Oct 30 23:37:32.675710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.675686 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5"] Oct 30 23:37:32.679765 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.679744 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 30 23:37:32.689541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.689474 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5"] Oct 30 23:37:32.690063 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.689997 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-843d8c-dockercfg-ptlb9\"" Oct 30 23:37:32.690309 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.690293 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-843d8c\"" Oct 30 23:37:32.698587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.698559 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg"] Oct 30 23:37:32.703138 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.703112 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 30 23:37:32.719681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.719619 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg"] Oct 30 23:37:32.803818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.803783 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k6hd\" (UniqueName: \"kubernetes.io/projected/c070c544-43c5-4796-b385-6c4094a63624-kube-api-access-7k6hd\") pod \"payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5\" (UID: \"c070c544-43c5-4796-b385-6c4094a63624\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 30 23:37:32.804002 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.803922 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rz9\" (UniqueName: \"kubernetes.io/projected/7f016eec-7091-4a3e-847e-7a382085ffef-kube-api-access-q6rz9\") pod \"payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg\" (UID: \"7f016eec-7091-4a3e-847e-7a382085ffef\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 30 23:37:32.804080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.804059 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f016eec-7091-4a3e-847e-7a382085ffef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg\" (UID: \"7f016eec-7091-4a3e-847e-7a382085ffef\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 30 23:37:32.804138 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.804108 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c070c544-43c5-4796-b385-6c4094a63624-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5\" (UID: \"c070c544-43c5-4796-b385-6c4094a63624\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 30 23:37:32.905210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.905161 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k6hd\" (UniqueName: \"kubernetes.io/projected/c070c544-43c5-4796-b385-6c4094a63624-kube-api-access-7k6hd\") pod \"payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5\" (UID: \"c070c544-43c5-4796-b385-6c4094a63624\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 30 23:37:32.905382 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.905252 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rz9\" (UniqueName: \"kubernetes.io/projected/7f016eec-7091-4a3e-847e-7a382085ffef-kube-api-access-q6rz9\") pod \"payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg\" (UID: \"7f016eec-7091-4a3e-847e-7a382085ffef\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 30 23:37:32.905382 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.905326 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f016eec-7091-4a3e-847e-7a382085ffef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg\" (UID: \"7f016eec-7091-4a3e-847e-7a382085ffef\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 30 23:37:32.905382 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.905361 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c070c544-43c5-4796-b385-6c4094a63624-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5\" (UID: \"c070c544-43c5-4796-b385-6c4094a63624\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 30 23:37:32.907611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.907543 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f016eec-7091-4a3e-847e-7a382085ffef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg\" (UID: \"7f016eec-7091-4a3e-847e-7a382085ffef\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 30 23:37:32.918053 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.917950 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c070c544-43c5-4796-b385-6c4094a63624-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5\" (UID: \"c070c544-43c5-4796-b385-6c4094a63624\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 30 23:37:32.932328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.931946 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k6hd\" (UniqueName: \"kubernetes.io/projected/c070c544-43c5-4796-b385-6c4094a63624-kube-api-access-7k6hd\") pod \"payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5\" (UID: \"c070c544-43c5-4796-b385-6c4094a63624\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 30 23:37:32.932328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.932295 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rz9\" (UniqueName: \"kubernetes.io/projected/7f016eec-7091-4a3e-847e-7a382085ffef-kube-api-access-q6rz9\") pod \"payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg\" (UID: \"7f016eec-7091-4a3e-847e-7a382085ffef\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 30 23:37:32.997779 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:32.997534 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 30 23:37:33.028511 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.028482 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm"] Oct 30 23:37:33.032416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.032393 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm"] Oct 30 23:37:33.032648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.032528 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 30 23:37:33.035056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.035031 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 30 23:37:33.036800 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.036778 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb"] Oct 30 23:37:33.041616 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.041588 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d3dcca\"" Oct 30 23:37:33.051526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.051501 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d3dcca-dockercfg-vk8p4\"" Oct 30 23:37:33.053071 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.053027 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb"] Oct 30 23:37:33.053196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.053164 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 30 23:37:33.123944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.114554 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6vx6\" (UniqueName: \"kubernetes.io/projected/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-kube-api-access-s6vx6\") pod \"payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb\" (UID: \"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 30 23:37:33.123944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.114714 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a6ea5-1c91-462f-94fd-7136a48971e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm\" (UID: \"1e8a6ea5-1c91-462f-94fd-7136a48971e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 30 23:37:33.123944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.114775 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4xqq\" (UniqueName: \"kubernetes.io/projected/1e8a6ea5-1c91-462f-94fd-7136a48971e9-kube-api-access-n4xqq\") pod \"payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm\" (UID: \"1e8a6ea5-1c91-462f-94fd-7136a48971e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 30 23:37:33.123944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.114809 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb\" (UID: \"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 30 23:37:33.217591 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.216614 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4xqq\" (UniqueName: \"kubernetes.io/projected/1e8a6ea5-1c91-462f-94fd-7136a48971e9-kube-api-access-n4xqq\") pod \"payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm\" (UID: \"1e8a6ea5-1c91-462f-94fd-7136a48971e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 30 23:37:33.217591 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.216678 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb\" (UID: \"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 30 23:37:33.217591 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.216822 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6vx6\" (UniqueName: \"kubernetes.io/projected/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-kube-api-access-s6vx6\") pod \"payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb\" (UID: \"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 30 23:37:33.217591 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.216890 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a6ea5-1c91-462f-94fd-7136a48971e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm\" (UID: \"1e8a6ea5-1c91-462f-94fd-7136a48971e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 30 23:37:33.218645 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.218315 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a6ea5-1c91-462f-94fd-7136a48971e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm\" (UID: \"1e8a6ea5-1c91-462f-94fd-7136a48971e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 30 23:37:33.219556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.219524 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb\" (UID: \"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 30 23:37:33.233609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.233554 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6vx6\" (UniqueName: \"kubernetes.io/projected/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-kube-api-access-s6vx6\") pod \"payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb\" (UID: \"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 30 23:37:33.236930 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.236908 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4xqq\" (UniqueName: \"kubernetes.io/projected/1e8a6ea5-1c91-462f-94fd-7136a48971e9-kube-api-access-n4xqq\") pod \"payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm\" (UID: \"1e8a6ea5-1c91-462f-94fd-7136a48971e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 30 23:37:33.332909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.332875 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5"] Oct 30 23:37:33.337995 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:37:33.337961 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc070c544_43c5_4796_b385_6c4094a63624.slice/crio-08aec93d09964ae0c9f4b6d3936b47ab5a93421436c7d84e63dc60c04c83941d WatchSource:0}: Error finding container 08aec93d09964ae0c9f4b6d3936b47ab5a93421436c7d84e63dc60c04c83941d: Status 404 returned error can't find the container with id 08aec93d09964ae0c9f4b6d3936b47ab5a93421436c7d84e63dc60c04c83941d Oct 30 23:37:33.347211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.347186 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg"] Oct 30 23:37:33.356952 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:37:33.356921 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f016eec_7091_4a3e_847e_7a382085ffef.slice/crio-27b924632dc05efc69e605c5f5d276f45d0883a183b55b13381ce1f0c2108f12 WatchSource:0}: Error finding container 27b924632dc05efc69e605c5f5d276f45d0883a183b55b13381ce1f0c2108f12: Status 404 returned error can't find the container with id 27b924632dc05efc69e605c5f5d276f45d0883a183b55b13381ce1f0c2108f12 Oct 30 23:37:33.370233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.370049 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 30 23:37:33.385392 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.385204 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 30 23:37:33.500663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.498091 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" event={"ID":"7f016eec-7091-4a3e-847e-7a382085ffef","Type":"ContainerStarted","Data":"27b924632dc05efc69e605c5f5d276f45d0883a183b55b13381ce1f0c2108f12"} Oct 30 23:37:33.500663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.499749 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" event={"ID":"522aa71b-47e4-44b1-825c-bf81305d21b4","Type":"ContainerStarted","Data":"77a6206811d38439424d8cfd4ae9850942b50cf9cc4e90a5ff525bd300babfd2"} Oct 30 23:37:33.501777 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.501192 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" event={"ID":"c070c544-43c5-4796-b385-6c4094a63624","Type":"ContainerStarted","Data":"08aec93d09964ae0c9f4b6d3936b47ab5a93421436c7d84e63dc60c04c83941d"} Oct 30 23:37:33.664258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.662505 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm"] Oct 30 23:37:33.667136 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:37:33.667086 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8a6ea5_1c91_462f_94fd_7136a48971e9.slice/crio-9d936ff83b731740c54fdf7e84d5ccb4049f748048aaeff41a588dc5b33e4a6b WatchSource:0}: Error finding container 9d936ff83b731740c54fdf7e84d5ccb4049f748048aaeff41a588dc5b33e4a6b: Status 404 returned error can't find the container with id 9d936ff83b731740c54fdf7e84d5ccb4049f748048aaeff41a588dc5b33e4a6b Oct 30 23:37:33.695149 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:33.695119 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb"] Oct 30 23:37:33.700842 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:37:33.699247 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa36aaf_a967_456c_a544_9a2bcc4ac7d3.slice/crio-0d2b0c3363576a16525c48f9615c79028ff46d82f4bc3276437a0d5e9c5a2d0e WatchSource:0}: Error finding container 0d2b0c3363576a16525c48f9615c79028ff46d82f4bc3276437a0d5e9c5a2d0e: Status 404 returned error can't find the container with id 0d2b0c3363576a16525c48f9615c79028ff46d82f4bc3276437a0d5e9c5a2d0e Oct 30 23:37:34.513075 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:34.513037 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" event={"ID":"1e8a6ea5-1c91-462f-94fd-7136a48971e9","Type":"ContainerStarted","Data":"9d936ff83b731740c54fdf7e84d5ccb4049f748048aaeff41a588dc5b33e4a6b"} Oct 30 23:37:34.516527 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:34.516494 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" event={"ID":"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3","Type":"ContainerStarted","Data":"0d2b0c3363576a16525c48f9615c79028ff46d82f4bc3276437a0d5e9c5a2d0e"} Oct 30 23:37:36.448493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.448291 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn"] Oct 30 23:37:36.460926 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.458174 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.477683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.471354 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn"] Oct 30 23:37:36.477683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.471665 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:37:36.477683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.471740 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:37:36.477683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.471864 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-843d8c-serving-cert\"" Oct 30 23:37:36.477683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.472773 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-59scr\"" Oct 30 23:37:36.477683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.472859 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 23:37:36.577516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.577454 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-metrics-certs\") pod \"maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.577701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.577600 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.577701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.577663 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2533ebe5-857f-4255-8c85-5a526fe1062b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.577811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.577694 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ll6v\" (UniqueName: \"kubernetes.io/projected/2533ebe5-857f-4255-8c85-5a526fe1062b-kube-api-access-7ll6v\") pod \"maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.679352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.679316 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.679658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.679637 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2533ebe5-857f-4255-8c85-5a526fe1062b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.679777 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.679763 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ll6v\" (UniqueName: \"kubernetes.io/projected/2533ebe5-857f-4255-8c85-5a526fe1062b-kube-api-access-7ll6v\") pod \"maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.680147 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.680130 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-metrics-certs\") pod \"maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.735236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.734916 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2533ebe5-857f-4255-8c85-5a526fe1062b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.743559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.743521 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.745530 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.745500 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-metrics-certs\") pod \"maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.756102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.746446 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ll6v\" (UniqueName: \"kubernetes.io/projected/2533ebe5-857f-4255-8c85-5a526fe1062b-kube-api-access-7ll6v\") pod \"maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:36.783896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:36.783857 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:37.271300 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:37.271211 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn"] Oct 30 23:37:37.581433 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:37.581391 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" event={"ID":"2533ebe5-857f-4255-8c85-5a526fe1062b","Type":"ContainerStarted","Data":"6273bf5bd11077cfe56efa75842ee377c1d94cd593224d7b771f06461a724cf9"} Oct 30 23:37:38.056874 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.056847 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z"] Oct 30 23:37:38.063244 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.063220 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.076763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.073596 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-9ef03a-serving-cert\"" Oct 30 23:37:38.076763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.076697 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z"] Oct 30 23:37:38.162688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.158914 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-metrics-certs\") pod \"maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.162688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.159009 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15181ee-3e9a-40c9-a029-9ee504a971fa-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.162688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.159035 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xggl\" (UniqueName: \"kubernetes.io/projected/c15181ee-3e9a-40c9-a029-9ee504a971fa-kube-api-access-5xggl\") pod \"maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.162688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.159063 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.260899 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.260863 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.261043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.260984 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-metrics-certs\") pod \"maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.261167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.261144 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15181ee-3e9a-40c9-a029-9ee504a971fa-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.261214 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.261199 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xggl\" (UniqueName: \"kubernetes.io/projected/c15181ee-3e9a-40c9-a029-9ee504a971fa-kube-api-access-5xggl\") pod \"maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.263284 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.263072 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15181ee-3e9a-40c9-a029-9ee504a971fa-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.266180 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.266090 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.270613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.269399 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-metrics-certs\") pod \"maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.277715 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.277692 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xggl\" (UniqueName: \"kubernetes.io/projected/c15181ee-3e9a-40c9-a029-9ee504a971fa-kube-api-access-5xggl\") pod \"maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.453025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.452992 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:38.673141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.673041 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" event={"ID":"c070c544-43c5-4796-b385-6c4094a63624","Type":"ContainerStarted","Data":"d2d0452178b2079a293c0673a10ebba578c88a88d0facf27ff846b2140edd577"} Oct 30 23:37:38.673141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.673103 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 30 23:37:38.680034 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.679248 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" event={"ID":"7371f1ff-163e-49b2-a4ea-5a9bf3a18215","Type":"ContainerStarted","Data":"e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c"} Oct 30 23:37:38.680034 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.679950 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:37:38.683623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.683051 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" event={"ID":"1e8a6ea5-1c91-462f-94fd-7136a48971e9","Type":"ContainerStarted","Data":"cc6af65b1d32bcfd65a99cdbd83d31bf7073c45b98d943e624c0632032ffcd9f"} Oct 30 23:37:38.683623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.683574 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 30 23:37:38.738539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.713035 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" event={"ID":"7f016eec-7091-4a3e-847e-7a382085ffef","Type":"ContainerStarted","Data":"65b340054ff204ea3e9089ea510d55bf4e3712e10df9dd29f7782872c7f0b071"} Oct 30 23:37:38.738539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.713875 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 30 23:37:38.747337 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.747259 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" podStartSLOduration=2.2909342600000002 podStartE2EDuration="6.747239146s" totalImagesPullingTime="4.456304886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:38.691972265 +0000 UTC m=+1070.210973356" watchObservedRunningTime="2026-10-30 23:37:38.747239146 +0000 UTC m=+1070.266240237" Oct 30 23:37:38.753558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.750041 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" podStartSLOduration=2.046685954 podStartE2EDuration="6.750024505s" totalImagesPullingTime="4.703338551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:38.739930122 +0000 UTC m=+1070.258931215" watchObservedRunningTime="2026-10-30 23:37:38.750024505 +0000 UTC m=+1070.269025596" Oct 30 23:37:38.757816 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.757162 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" event={"ID":"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3","Type":"ContainerStarted","Data":"c43dd4085f8264bf875fd779a96aad3200de0e9c03ed69d1979990a51aa50960"} Oct 30 23:37:38.757816 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.757769 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 30 23:37:38.763910 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.763863 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" podStartSLOduration=1.9922287779999999 podStartE2EDuration="7.763851225s" totalImagesPullingTime="5.771622447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:38.763525534 +0000 UTC m=+1070.282526625" watchObservedRunningTime="2026-10-30 23:37:38.763851225 +0000 UTC m=+1070.282852314" Oct 30 23:37:38.807486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.785727 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" podStartSLOduration=2.081632869 podStartE2EDuration="6.785708481s" totalImagesPullingTime="4.704075612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:38.783589806 +0000 UTC m=+1070.302590898" watchObservedRunningTime="2026-10-30 23:37:38.785708481 +0000 UTC m=+1070.304709571" Oct 30 23:37:38.825496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.810981 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" podStartSLOduration=1.201268846 podStartE2EDuration="5.810963187s" totalImagesPullingTime="4.609694341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:38.809018165 +0000 UTC m=+1070.328019256" watchObservedRunningTime="2026-10-30 23:37:38.810963187 +0000 UTC m=+1070.329964279" Oct 30 23:37:38.871805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:38.871779 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z"] Oct 30 23:37:38.875575 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:37:38.875535 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15181ee_3e9a_40c9_a029_9ee504a971fa.slice/crio-043ba69bac9eb513802acf2f68d765dd15d3a300b711c95a5953e2f3739941a0 WatchSource:0}: Error finding container 043ba69bac9eb513802acf2f68d765dd15d3a300b711c95a5953e2f3739941a0: Status 404 returned error can't find the container with id 043ba69bac9eb513802acf2f68d765dd15d3a300b711c95a5953e2f3739941a0 Oct 30 23:37:39.875624 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:39.875437 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" event={"ID":"c15181ee-3e9a-40c9-a029-9ee504a971fa","Type":"ContainerStarted","Data":"043ba69bac9eb513802acf2f68d765dd15d3a300b711c95a5953e2f3739941a0"} Oct 30 23:37:41.938634 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:41.938593 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" event={"ID":"522aa71b-47e4-44b1-825c-bf81305d21b4","Type":"ContainerStarted","Data":"bf345e06964e0f29e4f25c50b80cc58ae8f87f140017f2f23d4fa69b57312ece"} Oct 30 23:37:41.939642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:41.939584 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:37:41.962493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:41.961196 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" podStartSLOduration=1.665554541 podStartE2EDuration="9.961178656s" totalImagesPullingTime="8.295624115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:41.960297223 +0000 UTC m=+1073.479298306" watchObservedRunningTime="2026-10-30 23:37:41.961178656 +0000 UTC m=+1073.480179747" Oct 30 23:37:44.497677 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:44.497635 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:37:44.498211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:44.497717 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:37:44.498611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:44.498586 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:37:44.498723 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:44.498627 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:37:44.966907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:44.966852 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" event={"ID":"2533ebe5-857f-4255-8c85-5a526fe1062b","Type":"ContainerStarted","Data":"3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076"} Oct 30 23:37:44.967096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:44.967081 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:44.968989 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:44.968958 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" event={"ID":"c15181ee-3e9a-40c9-a029-9ee504a971fa","Type":"ContainerStarted","Data":"3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b"} Oct 30 23:37:44.969112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:44.969011 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:45.027539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.027453 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podStartSLOduration=2.116877708 podStartE2EDuration="9.027432641s" totalImagesPullingTime="6.910554933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:45.002746504 +0000 UTC m=+1076.521747597" watchObservedRunningTime="2026-10-30 23:37:45.027432641 +0000 UTC m=+1076.546433746" Oct 30 23:37:45.028673 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.028631 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podStartSLOduration=1.385942022 podStartE2EDuration="7.028620063s" totalImagesPullingTime="5.642678041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:45.02772018 +0000 UTC m=+1076.546721270" watchObservedRunningTime="2026-10-30 23:37:45.028620063 +0000 UTC m=+1076.547621165" Oct 30 23:37:45.977483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.977431 7570 generic.go:356] "Generic (PLEG): container finished" podID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerID="dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca" exitCode=0 Oct 30 23:37:45.978036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.977530 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" event={"ID":"e8fab750-eae1-4f10-b8f3-3db73b00a1f9","Type":"ContainerDied","Data":"dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca"} Oct 30 23:37:45.978117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.978071 7570 scope.go:122] "RemoveContainer" containerID="dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca" Oct 30 23:37:45.979505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.979482 7570 generic.go:356] "Generic (PLEG): container finished" podID="31585445-5877-4947-8df5-f30317948127" containerID="74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261" exitCode=0 Oct 30 23:37:45.979622 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.979558 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" event={"ID":"31585445-5877-4947-8df5-f30317948127","Type":"ContainerDied","Data":"74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261"} Oct 30 23:37:45.979690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.979616 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:45.979748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.979684 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:45.979748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.979733 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:45.980128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.980109 7570 scope.go:122] "RemoveContainer" containerID="74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261" Oct 30 23:37:45.981544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.981515 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:45.981641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.981590 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:45.981698 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:45.981635 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:47.031487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:47.031394 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" event={"ID":"e8fab750-eae1-4f10-b8f3-3db73b00a1f9","Type":"ContainerStarted","Data":"73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d"} Oct 30 23:37:47.071041 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:47.071004 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" event={"ID":"31585445-5877-4947-8df5-f30317948127","Type":"ContainerStarted","Data":"4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008"} Oct 30 23:37:47.794925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:47.794880 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:47.795331 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:47.795308 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 23:37:47.795551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:47.795510 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 23:37:47.893501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:47.889861 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:47.893501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:47.889984 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 23:37:47.893501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:47.890027 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 23:37:48.800488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:48.797898 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 23:37:48.800488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:48.797962 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 23:37:48.916488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:48.904540 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 23:37:48.916488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:48.904604 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 23:37:49.797489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:49.793937 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 23:37:49.797489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:49.794030 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 23:37:49.890185 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:49.889742 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 23:37:49.890185 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:49.889802 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 23:37:49.909431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:49.907663 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 30 23:37:50.800435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:50.799887 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 23:37:50.800435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:50.799951 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 23:37:50.892325 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:50.891869 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 23:37:50.892325 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:50.891933 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 23:37:50.934494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:50.933604 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 30 23:37:50.940332 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:50.939615 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 30 23:37:50.940702 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:50.940683 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 30 23:37:50.943426 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:50.943391 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:37:51.053676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:51.050845 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:37:51.053676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:51.052965 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:37:51.794008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:51.793973 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 23:37:51.794330 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:51.794277 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 23:37:51.889488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:51.888871 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 23:37:51.889488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:51.888932 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 23:37:52.960347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:52.960317 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:52.961628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:52.961605 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:52.962661 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:52.962604 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:37:53.026709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.025317 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:53.026903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.026827 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:53.027743 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.027617 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:37:53.671079 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.671040 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4"] Oct 30 23:37:53.680369 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.680323 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.684369 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.684339 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4"] Oct 30 23:37:53.684538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.684510 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"default-dockercfg-rzhfh\"" Oct 30 23:37:53.685741 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.685715 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"kube-root-ca.crt\"" Oct 30 23:37:53.686803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.686780 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"openshift-service-ca.crt\"" Oct 30 23:37:53.688610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.688592 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"simulator-w0-d3dcca-kserve-self-signed-certs\"" Oct 30 23:37:53.795304 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.795266 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-home\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.795304 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.795311 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cklbs\" (UniqueName: \"kubernetes.io/projected/08cd1e28-c399-4cba-951e-0210a978109f-kube-api-access-cklbs\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.795565 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.795335 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-model-cache\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.795565 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.795415 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08cd1e28-c399-4cba-951e-0210a978109f-tls-certs\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.795647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.795563 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-tmp-dir\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.795750 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.795733 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-dshm\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.896399 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.896363 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-dshm\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.896614 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.896452 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-home\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.896614 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.896502 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cklbs\" (UniqueName: \"kubernetes.io/projected/08cd1e28-c399-4cba-951e-0210a978109f-kube-api-access-cklbs\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.896614 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.896532 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-model-cache\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.896614 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.896556 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08cd1e28-c399-4cba-951e-0210a978109f-tls-certs\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.896829 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.896638 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-tmp-dir\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.896994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.896957 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-model-cache\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.897080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.897049 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-tmp-dir\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.897080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.897070 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-home\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.899106 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.899079 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-dshm\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.899620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.899599 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08cd1e28-c399-4cba-951e-0210a978109f-tls-certs\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.909195 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.909166 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cklbs\" (UniqueName: \"kubernetes.io/projected/08cd1e28-c399-4cba-951e-0210a978109f-kube-api-access-cklbs\") pod \"simulator-w0-d3dcca-kserve-74cd68d695-9s4b4\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:53.960296 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.960225 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:37:53.997534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:53.997505 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:54.289826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:54.289803 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4"] Oct 30 23:37:54.293190 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:37:54.293160 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08cd1e28_c399_4cba_951e_0210a978109f.slice/crio-edb5c8fc6341300547078829125541cec8424c3d470ed622d86410a12c92d45a WatchSource:0}: Error finding container edb5c8fc6341300547078829125541cec8424c3d470ed622d86410a12c92d45a: Status 404 returned error can't find the container with id edb5c8fc6341300547078829125541cec8424c3d470ed622d86410a12c92d45a Oct 30 23:37:55.189964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:55.189931 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" event={"ID":"08cd1e28-c399-4cba-951e-0210a978109f","Type":"ContainerStarted","Data":"46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5"} Oct 30 23:37:55.192404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:55.189977 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" event={"ID":"08cd1e28-c399-4cba-951e-0210a978109f","Type":"ContainerStarted","Data":"edb5c8fc6341300547078829125541cec8424c3d470ed622d86410a12c92d45a"} Oct 30 23:37:55.212341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:55.212282 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podStartSLOduration=2.212262398 podStartE2EDuration="2.212262398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:55.209787099 +0000 UTC m=+1086.728788217" watchObservedRunningTime="2026-10-30 23:37:55.212262398 +0000 UTC m=+1086.731263490" Oct 30 23:37:55.998414 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:55.998381 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:37:56.197442 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:56.197405 7570 generic.go:356] "Generic (PLEG): container finished" podID="a6f473a6-9517-4367-b935-294dbd28ceec" containerID="8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50" exitCode=0 Oct 30 23:37:56.197959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:56.197489 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" event={"ID":"a6f473a6-9517-4367-b935-294dbd28ceec","Type":"ContainerDied","Data":"8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50"} Oct 30 23:37:56.198110 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:56.198093 7570 scope.go:122] "RemoveContainer" containerID="8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50" Oct 30 23:37:56.199415 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:56.199384 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:56.199524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:56.199448 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:56.199576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:56.199524 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:37:57.216684 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.216639 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" event={"ID":"a6f473a6-9517-4367-b935-294dbd28ceec","Type":"ContainerStarted","Data":"ec286697b6cc7b25e983764c1e1a5728facf90554c840204c0c4c9e59148fe30"} Oct 30 23:37:57.240546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.240512 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:57.241899 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.241875 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 23:37:57.242023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.241931 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 23:37:57.527543 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.525280 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt"] Oct 30 23:37:57.533891 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.533862 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.538922 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.538722 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"openshift-service-ca.crt\"" Oct 30 23:37:57.539069 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.539034 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"kube-root-ca.crt\"" Oct 30 23:37:57.539324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.539300 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"default-dockercfg-mdbds\"" Oct 30 23:37:57.539649 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.539629 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"simulator-w1-611ef3-kserve-self-signed-certs\"" Oct 30 23:37:57.556328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.551428 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt"] Oct 30 23:37:57.647080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.647040 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-tmp-dir\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.647244 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.647224 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-dshm\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.647327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.647257 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw6ps\" (UniqueName: \"kubernetes.io/projected/00b04bdb-908f-4ced-b3d2-1627d96513b6-kube-api-access-xw6ps\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.647374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.647342 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00b04bdb-908f-4ced-b3d2-1627d96513b6-tls-certs\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.647423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.647403 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-model-cache\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.647572 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.647556 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-home\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.748796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.748755 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-tmp-dir\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.749008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.748889 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-dshm\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.749008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.748925 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw6ps\" (UniqueName: \"kubernetes.io/projected/00b04bdb-908f-4ced-b3d2-1627d96513b6-kube-api-access-xw6ps\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.749008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.748980 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00b04bdb-908f-4ced-b3d2-1627d96513b6-tls-certs\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.749008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.749008 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-model-cache\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.749228 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.749165 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-home\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.749629 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.749608 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-home\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.749857 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.749842 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-tmp-dir\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.750455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.750429 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-model-cache\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.752452 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.752423 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-dshm\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.753253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.753233 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00b04bdb-908f-4ced-b3d2-1627d96513b6-tls-certs\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.763709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.763678 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw6ps\" (UniqueName: \"kubernetes.io/projected/00b04bdb-908f-4ced-b3d2-1627d96513b6-kube-api-access-xw6ps\") pod \"simulator-w1-611ef3-kserve-8455bfc968-b9llt\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:57.852311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:57.852281 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:37:58.153483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:58.153443 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt"] Oct 30 23:37:58.155808 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:37:58.155781 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00b04bdb_908f_4ced_b3d2_1627d96513b6.slice/crio-a3db04a3b132ee5a580c3cf71cf87558dffcbf9a52c618d238d14851cecea71f WatchSource:0}: Error finding container a3db04a3b132ee5a580c3cf71cf87558dffcbf9a52c618d238d14851cecea71f: Status 404 returned error can't find the container with id a3db04a3b132ee5a580c3cf71cf87558dffcbf9a52c618d238d14851cecea71f Oct 30 23:37:58.223071 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:58.223035 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" event={"ID":"00b04bdb-908f-4ced-b3d2-1627d96513b6","Type":"ContainerStarted","Data":"a3db04a3b132ee5a580c3cf71cf87558dffcbf9a52c618d238d14851cecea71f"} Oct 30 23:37:58.240513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:58.240444 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 23:37:58.240637 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:58.240527 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 23:37:58.455579 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:58.455406 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:37:59.235484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:59.234762 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" event={"ID":"00b04bdb-908f-4ced-b3d2-1627d96513b6","Type":"ContainerStarted","Data":"3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65"} Oct 30 23:37:59.241483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:59.240081 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 23:37:59.241483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:59.240126 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 23:37:59.254935 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:59.254878 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podStartSLOduration=2.254859604 podStartE2EDuration="2.254859604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:59.253928844 +0000 UTC m=+1090.772929946" watchObservedRunningTime="2026-10-30 23:37:59.254859604 +0000 UTC m=+1090.773860696" Oct 30 23:37:59.442418 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:37:59.442376 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:37:59.852891 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:37:59.852855 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:38:00.239873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:00.239783 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 23:38:00.239873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:00.239843 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 23:38:01.239949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:01.239913 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 23:38:01.240533 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:01.239974 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 23:38:02.240793 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:02.240759 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 23:38:02.241297 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:02.240815 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 23:38:03.246548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:03.246517 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:38:03.247552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:03.247533 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:38:03.997856 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:03.997823 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:38:04.011706 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:04.011667 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:38:04.290067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:04.289980 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:38:06.728344 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.728304 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs"] Oct 30 23:38:06.732354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.732332 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.743108 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.743085 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs"] Oct 30 23:38:06.743342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.743322 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"simulator-w2-9ef03a-kserve-self-signed-certs\"" Oct 30 23:38:06.743431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.743404 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"openshift-service-ca.crt\"" Oct 30 23:38:06.743431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.743424 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"default-dockercfg-4fxkj\"" Oct 30 23:38:06.743564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.743538 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"kube-root-ca.crt\"" Oct 30 23:38:06.816169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.816120 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kpqz\" (UniqueName: \"kubernetes.io/projected/223c27af-605a-42a8-a29c-ae1fa2b2ace6-kube-api-access-7kpqz\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.816357 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.816188 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-home\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.816357 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.816321 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tls-certs\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.816453 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.816355 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-dshm\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.816565 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.816539 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-model-cache\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.816615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.816589 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tmp-dir\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.918405 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.918369 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kpqz\" (UniqueName: \"kubernetes.io/projected/223c27af-605a-42a8-a29c-ae1fa2b2ace6-kube-api-access-7kpqz\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.918648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.918432 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-home\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.918648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.918512 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tls-certs\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.918648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.918539 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-dshm\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.918648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.918647 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-model-cache\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.918852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.918688 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tmp-dir\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.918933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.918908 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-home\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.919019 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.919003 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-model-cache\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.919141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.919123 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tmp-dir\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.921082 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.921059 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-dshm\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.921863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.921844 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tls-certs\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:06.930929 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:06.930904 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kpqz\" (UniqueName: \"kubernetes.io/projected/223c27af-605a-42a8-a29c-ae1fa2b2ace6-kube-api-access-7kpqz\") pod \"simulator-w2-9ef03a-kserve-cdb8584b5-76gjs\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:07.048506 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:07.048394 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:07.330714 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:07.330683 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs"] Oct 30 23:38:07.332689 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:38:07.332664 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod223c27af_605a_42a8_a29c_ae1fa2b2ace6.slice/crio-bab552a50f2dfdd15ad55596a1c6215f8fd896fe4915884b568c766e72c1bf41 WatchSource:0}: Error finding container bab552a50f2dfdd15ad55596a1c6215f8fd896fe4915884b568c766e72c1bf41: Status 404 returned error can't find the container with id bab552a50f2dfdd15ad55596a1c6215f8fd896fe4915884b568c766e72c1bf41 Oct 30 23:38:07.853413 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:07.853380 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:38:07.866590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:07.866560 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:38:08.291589 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:08.291498 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" event={"ID":"223c27af-605a-42a8-a29c-ae1fa2b2ace6","Type":"ContainerStarted","Data":"a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1"} Oct 30 23:38:08.291589 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:08.291551 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" event={"ID":"223c27af-605a-42a8-a29c-ae1fa2b2ace6","Type":"ContainerStarted","Data":"bab552a50f2dfdd15ad55596a1c6215f8fd896fe4915884b568c766e72c1bf41"} Oct 30 23:38:08.303422 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:08.303397 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:38:08.314272 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:08.314199 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podStartSLOduration=2.314184787 podStartE2EDuration="2.314184787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:08.312375882 +0000 UTC m=+1099.831376973" watchObservedRunningTime="2026-10-30 23:38:08.314184787 +0000 UTC m=+1099.833185877" Oct 30 23:38:09.048517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:09.048484 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:17.049405 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:17.049371 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:17.061922 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:17.061904 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:17.398760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:17.398729 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:38:28.178755 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.178717 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl"] Oct 30 23:38:28.184138 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.184114 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.192730 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.192704 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"premium-w0-d3dcca-kserve-self-signed-certs\"" Oct 30 23:38:28.194809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.194785 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl"] Oct 30 23:38:28.370842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.370812 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-model-cache\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.371021 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.370861 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-home\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.371021 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.370962 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9smp\" (UniqueName: \"kubernetes.io/projected/3e57b4d8-7b73-4490-9138-de9401d6fd65-kube-api-access-x9smp\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.371093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.371020 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-dshm\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.371197 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.371176 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e57b4d8-7b73-4490-9138-de9401d6fd65-tls-certs\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.371241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.371231 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-tmp-dir\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.472416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.472334 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-tmp-dir\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.472416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.472377 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-model-cache\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.472416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.472413 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-home\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.472717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.472493 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9smp\" (UniqueName: \"kubernetes.io/projected/3e57b4d8-7b73-4490-9138-de9401d6fd65-kube-api-access-x9smp\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.472717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.472556 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-dshm\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.472821 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.472716 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e57b4d8-7b73-4490-9138-de9401d6fd65-tls-certs\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.472918 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.472836 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-tmp-dir\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.472918 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.472883 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-home\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.472918 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.472900 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-model-cache\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.475026 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.475002 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-dshm\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.475696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.475677 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e57b4d8-7b73-4490-9138-de9401d6fd65-tls-certs\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.485503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.485440 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9smp\" (UniqueName: \"kubernetes.io/projected/3e57b4d8-7b73-4490-9138-de9401d6fd65-kube-api-access-x9smp\") pod \"premium-w0-d3dcca-kserve-64955fd666-hx4zl\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.499186 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.499156 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:28.777749 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:28.777728 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl"] Oct 30 23:38:28.779814 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:38:28.779784 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e57b4d8_7b73_4490_9138_de9401d6fd65.slice/crio-12fc255c3bb7ece698a6f95273d1f617785a9d1dd3e15b0bbc2f9250f15c0550 WatchSource:0}: Error finding container 12fc255c3bb7ece698a6f95273d1f617785a9d1dd3e15b0bbc2f9250f15c0550: Status 404 returned error can't find the container with id 12fc255c3bb7ece698a6f95273d1f617785a9d1dd3e15b0bbc2f9250f15c0550 Oct 30 23:38:29.145640 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.145606 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f"] Oct 30 23:38:29.149783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.149765 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.155917 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.155220 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"openshift-service-ca.crt\"" Oct 30 23:38:29.156674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.156509 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"kube-root-ca.crt\"" Oct 30 23:38:29.156917 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.156671 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"default-dockercfg-jtbmr\"" Oct 30 23:38:29.156917 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.156799 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"premium-w3-b567bb-kserve-self-signed-certs\"" Oct 30 23:38:29.159328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.159309 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f"] Oct 30 23:38:29.282279 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.282172 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-dshm\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.282279 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.282249 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-model-cache\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.282925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.282351 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4nc\" (UniqueName: \"kubernetes.io/projected/ed4e6002-31dc-4eb3-9988-d808eee76c7f-kube-api-access-wj4nc\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.282925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.282400 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tmp-dir\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.282925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.282564 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tls-certs\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.282925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.282628 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-home\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.384266 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.384233 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tmp-dir\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.384445 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.384315 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tls-certs\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.384445 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.384353 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-home\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.384445 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.384381 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-dshm\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.384445 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.384424 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-model-cache\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.384646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.384567 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4nc\" (UniqueName: \"kubernetes.io/projected/ed4e6002-31dc-4eb3-9988-d808eee76c7f-kube-api-access-wj4nc\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.384718 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.384692 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tmp-dir\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.384780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.384745 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-home\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.384878 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.384855 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-model-cache\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.386911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.386886 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-dshm\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.387396 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.387376 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tls-certs\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.398051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.397959 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4nc\" (UniqueName: \"kubernetes.io/projected/ed4e6002-31dc-4eb3-9988-d808eee76c7f-kube-api-access-wj4nc\") pod \"premium-w3-b567bb-kserve-586f4c9b9-rn44f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.413033 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:38:29.412996 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:38:29.462800 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.462763 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:29.468784 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.468742 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" event={"ID":"3e57b4d8-7b73-4490-9138-de9401d6fd65","Type":"ContainerStarted","Data":"f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2"} Oct 30 23:38:29.468970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.468802 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" event={"ID":"3e57b4d8-7b73-4490-9138-de9401d6fd65","Type":"ContainerStarted","Data":"12fc255c3bb7ece698a6f95273d1f617785a9d1dd3e15b0bbc2f9250f15c0550"} Oct 30 23:38:29.493356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.493293 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podStartSLOduration=1.493271096 podStartE2EDuration="1.493271096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:29.490008919 +0000 UTC m=+1121.009010044" watchObservedRunningTime="2026-10-30 23:38:29.493271096 +0000 UTC m=+1121.012272187" Oct 30 23:38:29.499422 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.499392 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:29.749282 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:29.749061 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f"] Oct 30 23:38:29.752080 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:38:29.752047 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded4e6002_31dc_4eb3_9988_d808eee76c7f.slice/crio-be4405ad829b994ef067dcd8943fdf1c2e5174e4eb80110f94affedb0a1a9b59 WatchSource:0}: Error finding container be4405ad829b994ef067dcd8943fdf1c2e5174e4eb80110f94affedb0a1a9b59: Status 404 returned error can't find the container with id be4405ad829b994ef067dcd8943fdf1c2e5174e4eb80110f94affedb0a1a9b59 Oct 30 23:38:30.475744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:30.475711 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" event={"ID":"ed4e6002-31dc-4eb3-9988-d808eee76c7f","Type":"ContainerStarted","Data":"47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f"} Oct 30 23:38:30.475744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:30.475755 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" event={"ID":"ed4e6002-31dc-4eb3-9988-d808eee76c7f","Type":"ContainerStarted","Data":"be4405ad829b994ef067dcd8943fdf1c2e5174e4eb80110f94affedb0a1a9b59"} Oct 30 23:38:30.496102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:30.496059 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podStartSLOduration=1.496046248 podStartE2EDuration="1.496046248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:30.49574788 +0000 UTC m=+1122.014748973" watchObservedRunningTime="2026-10-30 23:38:30.496046248 +0000 UTC m=+1122.015047332" Oct 30 23:38:31.464128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:31.464094 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:38.499626 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:38.499590 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:38.511851 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:38.511832 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:38.538735 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:38.538702 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:38:39.463835 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:39.463798 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:39.476740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:39.476716 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:39.543960 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:39.543926 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:38:41.336018 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.335985 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd"] Oct 30 23:38:41.341765 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.341745 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.348171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.348145 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"openshift-service-ca.crt\"" Oct 30 23:38:41.349815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.349781 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"kube-root-ca.crt\"" Oct 30 23:38:41.350137 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.350113 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"default-dockercfg-75htb\"" Oct 30 23:38:41.350259 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.350212 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"premium-w4-843d8c-kserve-self-signed-certs\"" Oct 30 23:38:41.352065 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.352041 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd"] Oct 30 23:38:41.430142 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.430111 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1bda07-310d-43a8-a926-9004487b0034-tls-certs\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.430291 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.430149 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-model-cache\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.430348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.430329 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-tmp-dir\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.430429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.430399 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-home\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.430429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.430424 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-dshm\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.430600 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.430581 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wgvm\" (UniqueName: \"kubernetes.io/projected/4f1bda07-310d-43a8-a926-9004487b0034-kube-api-access-6wgvm\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.531338 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.531304 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-tmp-dir\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.531556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.531358 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-home\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.531556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.531384 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-dshm\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.531556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.531441 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wgvm\" (UniqueName: \"kubernetes.io/projected/4f1bda07-310d-43a8-a926-9004487b0034-kube-api-access-6wgvm\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.531556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.531523 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1bda07-310d-43a8-a926-9004487b0034-tls-certs\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.531556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.531547 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-model-cache\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.532115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.532087 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-home\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.532239 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.532159 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-tmp-dir\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.532287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.532264 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-model-cache\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.534283 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.534253 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1bda07-310d-43a8-a926-9004487b0034-tls-certs\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.534417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.534394 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-dshm\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.544052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.544024 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wgvm\" (UniqueName: \"kubernetes.io/projected/4f1bda07-310d-43a8-a926-9004487b0034-kube-api-access-6wgvm\") pod \"premium-w4-843d8c-kserve-7844d66896-27fxd\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.654724 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.654653 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:41.933388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:41.933364 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd"] Oct 30 23:38:41.935448 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:38:41.935416 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f1bda07_310d_43a8_a926_9004487b0034.slice/crio-9156f3c108220aa31db305d5ed5205f257011d4f57f8818644c3dc33791a5458 WatchSource:0}: Error finding container 9156f3c108220aa31db305d5ed5205f257011d4f57f8818644c3dc33791a5458: Status 404 returned error can't find the container with id 9156f3c108220aa31db305d5ed5205f257011d4f57f8818644c3dc33791a5458 Oct 30 23:38:42.560240 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:42.560200 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" event={"ID":"4f1bda07-310d-43a8-a926-9004487b0034","Type":"ContainerStarted","Data":"92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8"} Oct 30 23:38:42.560797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:42.560253 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" event={"ID":"4f1bda07-310d-43a8-a926-9004487b0034","Type":"ContainerStarted","Data":"9156f3c108220aa31db305d5ed5205f257011d4f57f8818644c3dc33791a5458"} Oct 30 23:38:42.585090 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:42.585045 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podStartSLOduration=1.585030683 podStartE2EDuration="1.585030683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:42.582932847 +0000 UTC m=+1134.101933940" watchObservedRunningTime="2026-10-30 23:38:42.585030683 +0000 UTC m=+1134.104031773" Oct 30 23:38:42.655416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:42.655383 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:51.655763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:51.655731 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:51.668600 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:51.668571 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:52.633079 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:52.633053 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:38:55.829804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:55.829770 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq"] Oct 30 23:38:55.835338 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:55.835319 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:55.839830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:55.839792 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-e15304-openshift-default-dockercfg-qbhbd\"" Oct 30 23:38:55.846037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:55.846017 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq"] Oct 30 23:38:56.032990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.032957 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.033164 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.033074 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67262faa-4e17-4e80-9d44-917ef37e70db-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.033164 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.033125 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67262faa-4e17-4e80-9d44-917ef37e70db-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.033242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.033171 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.033242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.033221 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.033305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.033287 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.033356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.033344 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2r47\" (UniqueName: \"kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-kube-api-access-b2r47\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.033417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.033406 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.033475 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.033449 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.134958 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.135014 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.135074 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67262faa-4e17-4e80-9d44-917ef37e70db-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.135112 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67262faa-4e17-4e80-9d44-917ef37e70db-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.135150 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.135173 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.135197 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.135274 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2r47\" (UniqueName: \"kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-kube-api-access-b2r47\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.135342 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.135394 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135737 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.135598 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.135765 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.135822 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.135795 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.136231 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.136207 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67262faa-4e17-4e80-9d44-917ef37e70db-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.137756 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.137724 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.137955 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.137937 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67262faa-4e17-4e80-9d44-917ef37e70db-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.147200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.147172 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2r47\" (UniqueName: \"kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-kube-api-access-b2r47\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.147309 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.147230 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.150226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.150205 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:56.435845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.435823 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq"] Oct 30 23:38:56.438090 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:38:56.438063 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67262faa_4e17_4e80_9d44_917ef37e70db.slice/crio-bb08f631d7bd8e1e107d53fa8ae79e6a1fa994d2dcea9671bb9d0804102e06bf WatchSource:0}: Error finding container bb08f631d7bd8e1e107d53fa8ae79e6a1fa994d2dcea9671bb9d0804102e06bf: Status 404 returned error can't find the container with id bb08f631d7bd8e1e107d53fa8ae79e6a1fa994d2dcea9671bb9d0804102e06bf Oct 30 23:38:56.440750 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.440711 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:38:56.440868 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.440792 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:38:56.440868 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.440840 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:38:56.646277 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.646232 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" event={"ID":"67262faa-4e17-4e80-9d44-917ef37e70db","Type":"ContainerStarted","Data":"5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357"} Oct 30 23:38:56.646483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.646289 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" event={"ID":"67262faa-4e17-4e80-9d44-917ef37e70db","Type":"ContainerStarted","Data":"bb08f631d7bd8e1e107d53fa8ae79e6a1fa994d2dcea9671bb9d0804102e06bf"} Oct 30 23:38:56.670542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:56.670455 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podStartSLOduration=1.670437219 podStartE2EDuration="1.670437219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:56.669932281 +0000 UTC m=+1148.188933373" watchObservedRunningTime="2026-10-30 23:38:56.670437219 +0000 UTC m=+1148.189438311" Oct 30 23:38:57.151402 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:57.151372 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:38:57.153074 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:57.153052 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 23:38:57.153192 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:57.153096 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 23:38:58.151537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:58.151501 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 23:38:58.151964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:58.151552 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 23:38:59.151009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:59.150973 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 23:38:59.151218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:38:59.151032 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 23:38:59.434919 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:38:59.434831 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:39:00.151131 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:00.151096 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 23:39:00.151328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:00.151148 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 23:39:01.151141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:01.151106 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 23:39:01.151588 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:01.151162 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 23:39:02.151142 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:02.151106 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 23:39:02.151364 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:02.151168 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 23:39:03.942840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:03.942804 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:39:03.944872 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:03.944850 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:39:03.954692 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:03.952934 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:39:13.766407 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:13.766376 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v"] Oct 30 23:39:13.770628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:13.770609 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:13.777660 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:13.777641 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:39:13.777759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:13.777732 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:39:13.783024 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:13.783001 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v"] Oct 30 23:39:13.908336 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:13.908308 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca170e97-t728v\" (UID: \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:13.908490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:13.908358 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxz5w\" (UniqueName: \"kubernetes.io/projected/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-kube-api-access-gxz5w\") pod \"maas-api-revoke-sub-ca170e97-t728v\" (UID: \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:13.908490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:13.908392 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca170e97-t728v\" (UID: \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:14.009709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:14.009689 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca170e97-t728v\" (UID: \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:14.009830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:14.009740 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxz5w\" (UniqueName: \"kubernetes.io/projected/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-kube-api-access-gxz5w\") pod \"maas-api-revoke-sub-ca170e97-t728v\" (UID: \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:14.009830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:14.009763 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca170e97-t728v\" (UID: \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:14.010399 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:14.010380 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca170e97-t728v\" (UID: \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:14.012068 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:14.012049 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca170e97-t728v\" (UID: \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:14.022659 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:14.022607 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxz5w\" (UniqueName: \"kubernetes.io/projected/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-kube-api-access-gxz5w\") pod \"maas-api-revoke-sub-ca170e97-t728v\" (UID: \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:14.085502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:14.085452 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:14.367617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:14.367594 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v"] Oct 30 23:39:14.369976 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:39:14.369950 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eba9836_e9a4_4043_a10d_a8a26eaa9ac9.slice/crio-106dd6bee350701f4d3dda6a5e88e55c3dd5d5208cc6a65fbcd2b920769f3c44 WatchSource:0}: Error finding container 106dd6bee350701f4d3dda6a5e88e55c3dd5d5208cc6a65fbcd2b920769f3c44: Status 404 returned error can't find the container with id 106dd6bee350701f4d3dda6a5e88e55c3dd5d5208cc6a65fbcd2b920769f3c44 Oct 30 23:39:14.371879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:14.371862 7570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:39:14.761141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:14.761055 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" event={"ID":"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9","Type":"ContainerStarted","Data":"106dd6bee350701f4d3dda6a5e88e55c3dd5d5208cc6a65fbcd2b920769f3c44"} Oct 30 23:39:18.785437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:18.785357 7570 generic.go:356] "Generic (PLEG): container finished" podID="8eba9836-e9a4-4043-a10d-a8a26eaa9ac9" containerID="29a9817cca5bb5fa13d295de95681b252e9fe6aeed0ec69971c22964829ee84d" exitCode=0 Oct 30 23:39:18.785437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:18.785427 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" event={"ID":"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9","Type":"ContainerDied","Data":"29a9817cca5bb5fa13d295de95681b252e9fe6aeed0ec69971c22964829ee84d"} Oct 30 23:39:19.924051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:19.924009 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:20.101602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:20.101568 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-xks-ca\") pod \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\" (UID: \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\") " Oct 30 23:39:20.101790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:20.101674 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-openshift-ca\") pod \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\" (UID: \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\") " Oct 30 23:39:20.101790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:20.101738 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-kube-api-access-gxz5w\" (UniqueName: \"kubernetes.io/projected/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-kube-api-access-gxz5w\") pod \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\" (UID: \"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9\") " Oct 30 23:39:20.102067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:20.102044 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-openshift-ca" pod "8eba9836-e9a4-4043-a10d-a8a26eaa9ac9" (UID: "8eba9836-e9a4-4043-a10d-a8a26eaa9ac9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:39:20.102191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:20.102176 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:20.103905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:20.103884 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-xks-ca" pod "8eba9836-e9a4-4043-a10d-a8a26eaa9ac9" (UID: "8eba9836-e9a4-4043-a10d-a8a26eaa9ac9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:39:20.104041 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:20.104028 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-kube-api-access-gxz5w" pod "8eba9836-e9a4-4043-a10d-a8a26eaa9ac9" (UID: "8eba9836-e9a4-4043-a10d-a8a26eaa9ac9"). InnerVolumeSpecName "kube-api-access-gxz5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:39:20.202892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:20.202866 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxz5w\" (UniqueName: \"kubernetes.io/projected/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-kube-api-access-gxz5w\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:20.202892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:20.202891 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:20.797529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:20.797495 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" event={"ID":"8eba9836-e9a4-4043-a10d-a8a26eaa9ac9","Type":"ContainerDied","Data":"106dd6bee350701f4d3dda6a5e88e55c3dd5d5208cc6a65fbcd2b920769f3c44"} Oct 30 23:39:20.797679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:20.797555 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="106dd6bee350701f4d3dda6a5e88e55c3dd5d5208cc6a65fbcd2b920769f3c44" Oct 30 23:39:20.797679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:20.797506 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v" Oct 30 23:39:25.480545 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.480512 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv"] Oct 30 23:39:25.481387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.481365 7570 cpu_manager.go:414] "containerMap: removing container" podUID="8eba9836-e9a4-4043-a10d-a8a26eaa9ac9" containerName="revoke-keys" Oct 30 23:39:25.481387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.481387 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eba9836-e9a4-4043-a10d-a8a26eaa9ac9" containerName="revoke-keys" Oct 30 23:39:25.481597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.481575 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eba9836-e9a4-4043-a10d-a8a26eaa9ac9" containerName="revoke-keys" Oct 30 23:39:25.485171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.485150 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.495894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.495870 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9ef03a-openshift-default-dockercfg-mf4wj\"" Oct 30 23:39:25.498627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.498601 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv"] Oct 30 23:39:25.544884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.544839 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs"] Oct 30 23:39:25.560099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.560068 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.561878 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.561851 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.562018 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.561891 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.562117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.561931 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.562216 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.562158 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.562674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.562630 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.562795 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.562693 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rqhk\" (UniqueName: \"kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-kube-api-access-7rqhk\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.562795 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.562784 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.562916 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.562865 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.562971 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.562923 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.565293 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.563609 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs"] Oct 30 23:39:25.595814 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.595788 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-843d8c-openshift-default-dockercfg-l4st5\"" Oct 30 23:39:25.664090 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664058 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.664090 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664085 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ms64\" (UniqueName: \"kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-kube-api-access-9ms64\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.664267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664120 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.664267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664187 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.664370 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664349 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.664425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664411 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.664502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664446 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.664547 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664528 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.664596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664557 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.664596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664588 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.664695 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664592 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.664750 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664693 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.664863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664747 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rqhk\" (UniqueName: \"kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-kube-api-access-7rqhk\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.664863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664767 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d62150f-cb32-4975-a88e-70b671493882-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.664863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664795 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d62150f-cb32-4975-a88e-70b671493882-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.664863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664820 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.665044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.664931 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.665044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.665027 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.665128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.665093 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.665128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.665078 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.665229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.665215 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.665271 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.665237 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.665271 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.665261 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.667413 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.667390 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.667666 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.667646 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.676925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.676903 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rqhk\" (UniqueName: \"kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-kube-api-access-7rqhk\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.677189 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.677173 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.765951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.765894 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.765951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.765929 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.765951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.765945 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ms64\" (UniqueName: \"kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-kube-api-access-9ms64\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.766198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.765972 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.766198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.766033 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.766198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.766092 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.766198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.766161 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d62150f-cb32-4975-a88e-70b671493882-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.766198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.766187 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d62150f-cb32-4975-a88e-70b671493882-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.766457 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.766242 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.766457 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.766383 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.766632 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.766506 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.766946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.766921 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d62150f-cb32-4975-a88e-70b671493882-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.767023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.766921 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.767063 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.767027 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.768257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.768233 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.768945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.768924 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d62150f-cb32-4975-a88e-70b671493882-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.778292 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.778257 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.778664 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.778644 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ms64\" (UniqueName: \"kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-kube-api-access-9ms64\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:25.802494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.802456 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:25.897918 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:25.897887 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:26.093149 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.093121 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv"] Oct 30 23:39:26.094081 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:39:26.094056 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed6a3e1_fd2f_49b8_9af6_5d9a49f9d3e9.slice/crio-741e6d04777b0e73aa758c809f60a4ac99f48d5d680e568ff6f397cd9aff0231 WatchSource:0}: Error finding container 741e6d04777b0e73aa758c809f60a4ac99f48d5d680e568ff6f397cd9aff0231: Status 404 returned error can't find the container with id 741e6d04777b0e73aa758c809f60a4ac99f48d5d680e568ff6f397cd9aff0231 Oct 30 23:39:26.096839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.096799 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:39:26.096924 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.096887 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:39:26.096963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.096931 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:39:26.193755 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.193722 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs"] Oct 30 23:39:26.197825 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.197785 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:39:26.197966 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.197882 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:39:26.197966 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.197929 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:39:26.844489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.840133 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" event={"ID":"3d62150f-cb32-4975-a88e-70b671493882","Type":"ContainerStarted","Data":"43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab"} Oct 30 23:39:26.844489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.840284 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" event={"ID":"3d62150f-cb32-4975-a88e-70b671493882","Type":"ContainerStarted","Data":"b32ce357d11afa23c6e344b5646bbf6cd2a06ed720268c0ca0151ed391b9e92a"} Oct 30 23:39:26.851401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.851366 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" event={"ID":"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9","Type":"ContainerStarted","Data":"2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614"} Oct 30 23:39:26.851599 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.851422 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" event={"ID":"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9","Type":"ContainerStarted","Data":"741e6d04777b0e73aa758c809f60a4ac99f48d5d680e568ff6f397cd9aff0231"} Oct 30 23:39:26.876519 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.873150 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" podStartSLOduration=1.873069111 podStartE2EDuration="1.873069111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:26.869838454 +0000 UTC m=+1178.388839556" watchObservedRunningTime="2026-10-30 23:39:26.873069111 +0000 UTC m=+1178.392070202" Oct 30 23:39:26.917487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.915559 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:26.917487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:26.916598 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" podStartSLOduration=1.916579541 podStartE2EDuration="1.916579541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:26.895406066 +0000 UTC m=+1178.414407160" watchObservedRunningTime="2026-10-30 23:39:26.916579541 +0000 UTC m=+1178.435580632" Oct 30 23:39:27.811542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:27.809533 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:27.811542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:27.809717 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 23:39:27.811542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:27.809761 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 23:39:27.909488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:27.901285 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 23:39:27.909488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:27.901374 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" podUID="3d62150f-cb32-4975-a88e-70b671493882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 23:39:28.803489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:28.802969 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 23:39:28.803489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:28.803044 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 23:39:28.906856 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:28.906823 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 23:39:28.907134 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:28.907089 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" podUID="3d62150f-cb32-4975-a88e-70b671493882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 23:39:29.545604 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:39:29.544652 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:39:29.808574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:29.807551 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 23:39:29.808574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:29.807617 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 23:39:29.902488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:29.901896 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 23:39:29.902488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:29.901968 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" podUID="3d62150f-cb32-4975-a88e-70b671493882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 23:39:30.811486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:30.810855 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 23:39:30.811486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:30.810930 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 23:39:30.902537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:30.902232 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 23:39:30.902537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:30.902307 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" podUID="3d62150f-cb32-4975-a88e-70b671493882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 23:39:31.810487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:31.809924 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 23:39:31.810487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:31.810001 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 23:39:31.917487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:31.917011 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 23:39:31.917487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:31.917081 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" podUID="3d62150f-cb32-4975-a88e-70b671493882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 23:39:32.076723 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.076642 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n"] Oct 30 23:39:32.083045 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.083022 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n"] Oct 30 23:39:32.083296 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.083279 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:32.108487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.101262 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:39:32.109654 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.109109 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:39:32.265922 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.265881 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzrd\" (UniqueName: \"kubernetes.io/projected/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-kube-api-access-lnzrd\") pod \"maas-api-revoke-sub-579e0042-7247n\" (UID: \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:32.266378 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.266354 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-579e0042-7247n\" (UID: \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:32.266570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.266548 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-579e0042-7247n\" (UID: \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:32.368355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.368320 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-579e0042-7247n\" (UID: \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:32.368770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.368744 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzrd\" (UniqueName: \"kubernetes.io/projected/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-kube-api-access-lnzrd\") pod \"maas-api-revoke-sub-579e0042-7247n\" (UID: \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:32.369091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.369068 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-579e0042-7247n\" (UID: \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:32.369997 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.369974 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-579e0042-7247n\" (UID: \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:32.376483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.371036 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-579e0042-7247n\" (UID: \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:32.384342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.384319 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzrd\" (UniqueName: \"kubernetes.io/projected/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-kube-api-access-lnzrd\") pod \"maas-api-revoke-sub-579e0042-7247n\" (UID: \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:32.465788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.465759 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:32.763496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.760442 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n"] Oct 30 23:39:32.766228 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:39:32.766198 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2e7efb_89d5_4338_9a28_501fcc8b18dc.slice/crio-cae65c0da5e10a1d220b7ca4dab6a65523ef358ef246744c3a2b46d632e89372 WatchSource:0}: Error finding container cae65c0da5e10a1d220b7ca4dab6a65523ef358ef246744c3a2b46d632e89372: Status 404 returned error can't find the container with id cae65c0da5e10a1d220b7ca4dab6a65523ef358ef246744c3a2b46d632e89372 Oct 30 23:39:32.804571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.804539 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 23:39:32.804781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.804750 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 23:39:32.898825 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.898789 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 23:39:32.898961 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.898852 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" podUID="3d62150f-cb32-4975-a88e-70b671493882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 23:39:32.956879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:32.956844 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" event={"ID":"8a2e7efb-89d5-4338-9a28-501fcc8b18dc","Type":"ContainerStarted","Data":"cae65c0da5e10a1d220b7ca4dab6a65523ef358ef246744c3a2b46d632e89372"} Oct 30 23:39:33.808598 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:33.808560 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:33.809036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:33.809013 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:33.809897 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:33.809875 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:39:33.902818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:33.902792 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:33.903186 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:33.903157 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:33.904044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:33.904029 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:39:33.991083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:33.990999 7570 generic.go:356] "Generic (PLEG): container finished" podID="8a2e7efb-89d5-4338-9a28-501fcc8b18dc" containerID="6f28d35b39dede69c68bac0f441955dd9ad9bda7fd4ace7b3abe801b272bf5d2" exitCode=0 Oct 30 23:39:33.992856 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:33.992832 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" event={"ID":"8a2e7efb-89d5-4338-9a28-501fcc8b18dc","Type":"ContainerDied","Data":"6f28d35b39dede69c68bac0f441955dd9ad9bda7fd4ace7b3abe801b272bf5d2"} Oct 30 23:39:35.175677 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.175653 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:35.224623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.224163 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-kube-api-access-lnzrd\" (UniqueName: \"kubernetes.io/projected/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-kube-api-access-lnzrd\") pod \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\" (UID: \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\") " Oct 30 23:39:35.224623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.224496 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-openshift-ca\") pod \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\" (UID: \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\") " Oct 30 23:39:35.224984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.224743 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-xks-ca\") pod \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\" (UID: \"8a2e7efb-89d5-4338-9a28-501fcc8b18dc\") " Oct 30 23:39:35.225419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.225391 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-openshift-ca" pod "8a2e7efb-89d5-4338-9a28-501fcc8b18dc" (UID: "8a2e7efb-89d5-4338-9a28-501fcc8b18dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:39:35.225935 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.225905 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:35.231977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.231949 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-kube-api-access-lnzrd" pod "8a2e7efb-89d5-4338-9a28-501fcc8b18dc" (UID: "8a2e7efb-89d5-4338-9a28-501fcc8b18dc"). InnerVolumeSpecName "kube-api-access-lnzrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:39:35.233028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.233002 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-xks-ca" pod "8a2e7efb-89d5-4338-9a28-501fcc8b18dc" (UID: "8a2e7efb-89d5-4338-9a28-501fcc8b18dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:39:35.326838 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.326773 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:35.326838 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.326799 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnzrd\" (UniqueName: \"kubernetes.io/projected/8a2e7efb-89d5-4338-9a28-501fcc8b18dc-kube-api-access-lnzrd\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:35.372388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.372361 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf"] Oct 30 23:39:35.372984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.372969 7570 cpu_manager.go:414] "containerMap: removing container" podUID="8a2e7efb-89d5-4338-9a28-501fcc8b18dc" containerName="revoke-keys" Oct 30 23:39:35.373032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.372986 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2e7efb-89d5-4338-9a28-501fcc8b18dc" containerName="revoke-keys" Oct 30 23:39:35.373117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.373108 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2e7efb-89d5-4338-9a28-501fcc8b18dc" containerName="revoke-keys" Oct 30 23:39:35.376603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.376586 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:35.385896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.385872 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf"] Oct 30 23:39:35.428339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.428310 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98cee74f-mqcpf\" (UID: \"c8739891-ce63-4955-8382-74ceddeb4fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:35.428599 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.428578 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98cee74f-mqcpf\" (UID: \"c8739891-ce63-4955-8382-74ceddeb4fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:35.428671 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.428612 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9ps\" (UniqueName: \"kubernetes.io/projected/c8739891-ce63-4955-8382-74ceddeb4fc5-kube-api-access-gh9ps\") pod \"maas-api-revoke-sub-98cee74f-mqcpf\" (UID: \"c8739891-ce63-4955-8382-74ceddeb4fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:35.529215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.529188 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98cee74f-mqcpf\" (UID: \"c8739891-ce63-4955-8382-74ceddeb4fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:35.529334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.529223 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9ps\" (UniqueName: \"kubernetes.io/projected/c8739891-ce63-4955-8382-74ceddeb4fc5-kube-api-access-gh9ps\") pod \"maas-api-revoke-sub-98cee74f-mqcpf\" (UID: \"c8739891-ce63-4955-8382-74ceddeb4fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:35.529334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.529255 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98cee74f-mqcpf\" (UID: \"c8739891-ce63-4955-8382-74ceddeb4fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:35.529977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.529955 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98cee74f-mqcpf\" (UID: \"c8739891-ce63-4955-8382-74ceddeb4fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:35.531745 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.531725 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98cee74f-mqcpf\" (UID: \"c8739891-ce63-4955-8382-74ceddeb4fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:35.541583 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.541562 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9ps\" (UniqueName: \"kubernetes.io/projected/c8739891-ce63-4955-8382-74ceddeb4fc5-kube-api-access-gh9ps\") pod \"maas-api-revoke-sub-98cee74f-mqcpf\" (UID: \"c8739891-ce63-4955-8382-74ceddeb4fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:35.692018 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.691988 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:35.988559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:35.988531 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf"] Oct 30 23:39:35.989378 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:39:35.989354 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8739891_ce63_4955_8382_74ceddeb4fc5.slice/crio-c31b471db0a9769296c822e4e0467b82b349b064ef99eea08e397d9339913726 WatchSource:0}: Error finding container c31b471db0a9769296c822e4e0467b82b349b064ef99eea08e397d9339913726: Status 404 returned error can't find the container with id c31b471db0a9769296c822e4e0467b82b349b064ef99eea08e397d9339913726 Oct 30 23:39:36.006868 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:36.006834 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" event={"ID":"c8739891-ce63-4955-8382-74ceddeb4fc5","Type":"ContainerStarted","Data":"c31b471db0a9769296c822e4e0467b82b349b064ef99eea08e397d9339913726"} Oct 30 23:39:36.008532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:36.008509 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" event={"ID":"8a2e7efb-89d5-4338-9a28-501fcc8b18dc","Type":"ContainerDied","Data":"cae65c0da5e10a1d220b7ca4dab6a65523ef358ef246744c3a2b46d632e89372"} Oct 30 23:39:36.008638 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:36.008550 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cae65c0da5e10a1d220b7ca4dab6a65523ef358ef246744c3a2b46d632e89372" Oct 30 23:39:36.008691 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:36.008642 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n" Oct 30 23:39:37.016241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:37.016202 7570 generic.go:356] "Generic (PLEG): container finished" podID="c8739891-ce63-4955-8382-74ceddeb4fc5" containerID="8a244a5fcef2af70eb6c7f0ba666ef14a2e3a63d3c6f09f7a9e3bdbfb24a7d48" exitCode=0 Oct 30 23:39:37.016714 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:37.016406 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" event={"ID":"c8739891-ce63-4955-8382-74ceddeb4fc5","Type":"ContainerDied","Data":"8a244a5fcef2af70eb6c7f0ba666ef14a2e3a63d3c6f09f7a9e3bdbfb24a7d48"} Oct 30 23:39:38.162431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:38.162405 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:38.290383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:38.290311 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-openshift-ca\") pod \"c8739891-ce63-4955-8382-74ceddeb4fc5\" (UID: \"c8739891-ce63-4955-8382-74ceddeb4fc5\") " Oct 30 23:39:38.290383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:38.290371 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-xks-ca\") pod \"c8739891-ce63-4955-8382-74ceddeb4fc5\" (UID: \"c8739891-ce63-4955-8382-74ceddeb4fc5\") " Oct 30 23:39:38.290606 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:38.290445 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8739891-ce63-4955-8382-74ceddeb4fc5-kube-api-access-gh9ps\" (UniqueName: \"kubernetes.io/projected/c8739891-ce63-4955-8382-74ceddeb4fc5-kube-api-access-gh9ps\") pod \"c8739891-ce63-4955-8382-74ceddeb4fc5\" (UID: \"c8739891-ce63-4955-8382-74ceddeb4fc5\") " Oct 30 23:39:38.290772 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:38.290741 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-openshift-ca" pod "c8739891-ce63-4955-8382-74ceddeb4fc5" (UID: "c8739891-ce63-4955-8382-74ceddeb4fc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:39:38.291073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:38.291046 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:38.292949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:38.292920 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8739891-ce63-4955-8382-74ceddeb4fc5-kube-api-access-gh9ps" pod "c8739891-ce63-4955-8382-74ceddeb4fc5" (UID: "c8739891-ce63-4955-8382-74ceddeb4fc5"). InnerVolumeSpecName "kube-api-access-gh9ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:39:38.293050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:38.292959 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-xks-ca" pod "c8739891-ce63-4955-8382-74ceddeb4fc5" (UID: "c8739891-ce63-4955-8382-74ceddeb4fc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:39:38.392322 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:38.392293 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8739891-ce63-4955-8382-74ceddeb4fc5-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:38.392322 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:38.392320 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh9ps\" (UniqueName: \"kubernetes.io/projected/c8739891-ce63-4955-8382-74ceddeb4fc5-kube-api-access-gh9ps\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:39.031030 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:39.030962 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" Oct 30 23:39:39.031169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:39.030963 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf" event={"ID":"c8739891-ce63-4955-8382-74ceddeb4fc5","Type":"ContainerDied","Data":"c31b471db0a9769296c822e4e0467b82b349b064ef99eea08e397d9339913726"} Oct 30 23:39:39.031169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:39.031085 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c31b471db0a9769296c822e4e0467b82b349b064ef99eea08e397d9339913726" Oct 30 23:39:41.323671 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.323631 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq"] Oct 30 23:39:41.324226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.324210 7570 cpu_manager.go:414] "containerMap: removing container" podUID="c8739891-ce63-4955-8382-74ceddeb4fc5" containerName="revoke-keys" Oct 30 23:39:41.324270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.324227 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8739891-ce63-4955-8382-74ceddeb4fc5" containerName="revoke-keys" Oct 30 23:39:41.324368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.324359 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8739891-ce63-4955-8382-74ceddeb4fc5" containerName="revoke-keys" Oct 30 23:39:41.328084 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.328070 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:41.333906 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.333882 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:39:41.334080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.333975 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:39:41.337824 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.337790 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq"] Oct 30 23:39:41.455688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.455653 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a69287be-phqlq\" (UID: \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:41.455892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.455706 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm8w2\" (UniqueName: \"kubernetes.io/projected/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-kube-api-access-xm8w2\") pod \"maas-api-revoke-sub-a69287be-phqlq\" (UID: \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:41.455892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.455881 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a69287be-phqlq\" (UID: \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:41.557389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.557361 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a69287be-phqlq\" (UID: \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:41.557555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.557414 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm8w2\" (UniqueName: \"kubernetes.io/projected/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-kube-api-access-xm8w2\") pod \"maas-api-revoke-sub-a69287be-phqlq\" (UID: \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:41.557555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.557507 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a69287be-phqlq\" (UID: \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:41.558151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.558084 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a69287be-phqlq\" (UID: \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:41.559942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.559914 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a69287be-phqlq\" (UID: \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:41.569978 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.569915 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm8w2\" (UniqueName: \"kubernetes.io/projected/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-kube-api-access-xm8w2\") pod \"maas-api-revoke-sub-a69287be-phqlq\" (UID: \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:41.644423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.644402 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:41.921820 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:41.921789 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq"] Oct 30 23:39:41.924375 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:39:41.924345 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc200d5_c627_4beb_959c_0f0a4aaa60c1.slice/crio-6cbccd5a8ce81b9732dba8333f240c9fa83e5859d5f15512b267f0915567f8ae WatchSource:0}: Error finding container 6cbccd5a8ce81b9732dba8333f240c9fa83e5859d5f15512b267f0915567f8ae: Status 404 returned error can't find the container with id 6cbccd5a8ce81b9732dba8333f240c9fa83e5859d5f15512b267f0915567f8ae Oct 30 23:39:42.051641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:42.051606 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" event={"ID":"ffc200d5-c627-4beb-959c-0f0a4aaa60c1","Type":"ContainerStarted","Data":"6cbccd5a8ce81b9732dba8333f240c9fa83e5859d5f15512b267f0915567f8ae"} Oct 30 23:39:43.059054 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.058968 7570 generic.go:356] "Generic (PLEG): container finished" podID="ffc200d5-c627-4beb-959c-0f0a4aaa60c1" containerID="eae2854ae8e9419f12929a59fa257016f315996d53469ade0938a8acdef2879d" exitCode=0 Oct 30 23:39:43.059433 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.059061 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" event={"ID":"ffc200d5-c627-4beb-959c-0f0a4aaa60c1","Type":"ContainerDied","Data":"eae2854ae8e9419f12929a59fa257016f315996d53469ade0938a8acdef2879d"} Oct 30 23:39:43.561854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.561822 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc"] Oct 30 23:39:43.566351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.566336 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:43.575188 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.575163 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc"] Oct 30 23:39:43.688740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.688705 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1224a90c-bkrkc\" (UID: \"2aa07e4b-bd11-4810-af05-c8d358136fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:43.688895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.688877 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1224a90c-bkrkc\" (UID: \"2aa07e4b-bd11-4810-af05-c8d358136fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:43.688938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.688921 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4rhg\" (UniqueName: \"kubernetes.io/projected/2aa07e4b-bd11-4810-af05-c8d358136fe9-kube-api-access-q4rhg\") pod \"maas-api-revoke-sub-1224a90c-bkrkc\" (UID: \"2aa07e4b-bd11-4810-af05-c8d358136fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:43.789713 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.789685 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4rhg\" (UniqueName: \"kubernetes.io/projected/2aa07e4b-bd11-4810-af05-c8d358136fe9-kube-api-access-q4rhg\") pod \"maas-api-revoke-sub-1224a90c-bkrkc\" (UID: \"2aa07e4b-bd11-4810-af05-c8d358136fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:43.789875 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.789759 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1224a90c-bkrkc\" (UID: \"2aa07e4b-bd11-4810-af05-c8d358136fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:43.789875 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.789847 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1224a90c-bkrkc\" (UID: \"2aa07e4b-bd11-4810-af05-c8d358136fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:43.790434 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.790416 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1224a90c-bkrkc\" (UID: \"2aa07e4b-bd11-4810-af05-c8d358136fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:43.792227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.792206 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1224a90c-bkrkc\" (UID: \"2aa07e4b-bd11-4810-af05-c8d358136fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:43.802625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.802600 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4rhg\" (UniqueName: \"kubernetes.io/projected/2aa07e4b-bd11-4810-af05-c8d358136fe9-kube-api-access-q4rhg\") pod \"maas-api-revoke-sub-1224a90c-bkrkc\" (UID: \"2aa07e4b-bd11-4810-af05-c8d358136fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:43.881732 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:43.881710 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:44.180566 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:44.180534 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc"] Oct 30 23:39:44.182614 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:39:44.182583 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa07e4b_bd11_4810_af05_c8d358136fe9.slice/crio-15b66026f44c83a6dbad643139d3f3a32cb95550f725c59aef05817f45f33134 WatchSource:0}: Error finding container 15b66026f44c83a6dbad643139d3f3a32cb95550f725c59aef05817f45f33134: Status 404 returned error can't find the container with id 15b66026f44c83a6dbad643139d3f3a32cb95550f725c59aef05817f45f33134 Oct 30 23:39:44.213078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:44.213055 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:44.300856 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:44.300824 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-openshift-ca\") pod \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\" (UID: \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\") " Oct 30 23:39:44.301035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:44.301022 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-xks-ca\") pod \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\" (UID: \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\") " Oct 30 23:39:44.301110 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:44.301096 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-kube-api-access-xm8w2\" (UniqueName: \"kubernetes.io/projected/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-kube-api-access-xm8w2\") pod \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\" (UID: \"ffc200d5-c627-4beb-959c-0f0a4aaa60c1\") " Oct 30 23:39:44.301283 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:44.301255 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-openshift-ca" pod "ffc200d5-c627-4beb-959c-0f0a4aaa60c1" (UID: "ffc200d5-c627-4beb-959c-0f0a4aaa60c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:39:44.301601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:44.301580 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:44.386923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:44.386885 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-kube-api-access-xm8w2" pod "ffc200d5-c627-4beb-959c-0f0a4aaa60c1" (UID: "ffc200d5-c627-4beb-959c-0f0a4aaa60c1"). InnerVolumeSpecName "kube-api-access-xm8w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:39:44.386923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:44.386902 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-xks-ca" pod "ffc200d5-c627-4beb-959c-0f0a4aaa60c1" (UID: "ffc200d5-c627-4beb-959c-0f0a4aaa60c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:39:44.408495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:44.407643 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:44.408495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:44.407673 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm8w2\" (UniqueName: \"kubernetes.io/projected/ffc200d5-c627-4beb-959c-0f0a4aaa60c1-kube-api-access-xm8w2\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:45.075610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.075508 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" event={"ID":"2aa07e4b-bd11-4810-af05-c8d358136fe9","Type":"ContainerStarted","Data":"9b1857b3027e8bf6ab0b43fb00ba8bfa7787f6f706adbb5731c02d4bf0f4072d"} Oct 30 23:39:45.075610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.075564 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" event={"ID":"2aa07e4b-bd11-4810-af05-c8d358136fe9","Type":"ContainerStarted","Data":"15b66026f44c83a6dbad643139d3f3a32cb95550f725c59aef05817f45f33134"} Oct 30 23:39:45.076925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.076902 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" event={"ID":"ffc200d5-c627-4beb-959c-0f0a4aaa60c1","Type":"ContainerDied","Data":"6cbccd5a8ce81b9732dba8333f240c9fa83e5859d5f15512b267f0915567f8ae"} Oct 30 23:39:45.077061 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.076941 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cbccd5a8ce81b9732dba8333f240c9fa83e5859d5f15512b267f0915567f8ae" Oct 30 23:39:45.077061 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.076949 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq" Oct 30 23:39:45.094311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.094263 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" podStartSLOduration=2.094247564 podStartE2EDuration="2.094247564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:45.093134764 +0000 UTC m=+1196.612135855" watchObservedRunningTime="2026-10-30 23:39:45.094247564 +0000 UTC m=+1196.613248656" Oct 30 23:39:45.684174 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.684143 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q"] Oct 30 23:39:45.684736 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.684722 7570 cpu_manager.go:414] "containerMap: removing container" podUID="ffc200d5-c627-4beb-959c-0f0a4aaa60c1" containerName="revoke-keys" Oct 30 23:39:45.684787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.684737 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc200d5-c627-4beb-959c-0f0a4aaa60c1" containerName="revoke-keys" Oct 30 23:39:45.684870 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.684860 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc200d5-c627-4beb-959c-0f0a4aaa60c1" containerName="revoke-keys" Oct 30 23:39:45.688307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.688292 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:45.697342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.697320 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q"] Oct 30 23:39:45.824490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.823432 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659ff0c5-5gc8q\" (UID: \"0a2cbe38-7fc7-4274-8834-375b73af158a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:45.824490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.823586 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659ff0c5-5gc8q\" (UID: \"0a2cbe38-7fc7-4274-8834-375b73af158a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:45.824490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.823674 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9wqr\" (UniqueName: \"kubernetes.io/projected/0a2cbe38-7fc7-4274-8834-375b73af158a-kube-api-access-j9wqr\") pod \"maas-api-revoke-sub-659ff0c5-5gc8q\" (UID: \"0a2cbe38-7fc7-4274-8834-375b73af158a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:45.924666 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.924633 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9wqr\" (UniqueName: \"kubernetes.io/projected/0a2cbe38-7fc7-4274-8834-375b73af158a-kube-api-access-j9wqr\") pod \"maas-api-revoke-sub-659ff0c5-5gc8q\" (UID: \"0a2cbe38-7fc7-4274-8834-375b73af158a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:45.924833 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.924743 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659ff0c5-5gc8q\" (UID: \"0a2cbe38-7fc7-4274-8834-375b73af158a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:45.924833 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.924794 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659ff0c5-5gc8q\" (UID: \"0a2cbe38-7fc7-4274-8834-375b73af158a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:45.925396 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.925378 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659ff0c5-5gc8q\" (UID: \"0a2cbe38-7fc7-4274-8834-375b73af158a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:45.927232 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.927215 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659ff0c5-5gc8q\" (UID: \"0a2cbe38-7fc7-4274-8834-375b73af158a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:45.937076 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:45.937016 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9wqr\" (UniqueName: \"kubernetes.io/projected/0a2cbe38-7fc7-4274-8834-375b73af158a-kube-api-access-j9wqr\") pod \"maas-api-revoke-sub-659ff0c5-5gc8q\" (UID: \"0a2cbe38-7fc7-4274-8834-375b73af158a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:46.003751 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:46.003716 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:46.089494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:46.088096 7570 generic.go:356] "Generic (PLEG): container finished" podID="2aa07e4b-bd11-4810-af05-c8d358136fe9" containerID="9b1857b3027e8bf6ab0b43fb00ba8bfa7787f6f706adbb5731c02d4bf0f4072d" exitCode=0 Oct 30 23:39:46.089494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:46.088147 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" event={"ID":"2aa07e4b-bd11-4810-af05-c8d358136fe9","Type":"ContainerDied","Data":"9b1857b3027e8bf6ab0b43fb00ba8bfa7787f6f706adbb5731c02d4bf0f4072d"} Oct 30 23:39:46.324656 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:46.324577 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q"] Oct 30 23:39:46.328982 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:39:46.328952 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2cbe38_7fc7_4274_8834_375b73af158a.slice/crio-63c8bb8b0d2286b3753f13dba2fb93cf97e772bf1ab2eed5d6b9b0edbf1fb461 WatchSource:0}: Error finding container 63c8bb8b0d2286b3753f13dba2fb93cf97e772bf1ab2eed5d6b9b0edbf1fb461: Status 404 returned error can't find the container with id 63c8bb8b0d2286b3753f13dba2fb93cf97e772bf1ab2eed5d6b9b0edbf1fb461 Oct 30 23:39:47.097150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.097073 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" event={"ID":"0a2cbe38-7fc7-4274-8834-375b73af158a","Type":"ContainerStarted","Data":"a3349f1ce336672604ffe9559c69a2151851d01ef5f285d5a2a6a0ae543db807"} Oct 30 23:39:47.097666 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.097182 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" event={"ID":"0a2cbe38-7fc7-4274-8834-375b73af158a","Type":"ContainerStarted","Data":"63c8bb8b0d2286b3753f13dba2fb93cf97e772bf1ab2eed5d6b9b0edbf1fb461"} Oct 30 23:39:47.124765 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.124708 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" podStartSLOduration=2.124694233 podStartE2EDuration="2.124694233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:47.12049079 +0000 UTC m=+1198.639491871" watchObservedRunningTime="2026-10-30 23:39:47.124694233 +0000 UTC m=+1198.643695322" Oct 30 23:39:47.253262 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.253240 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:47.357157 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.357089 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-openshift-ca\") pod \"2aa07e4b-bd11-4810-af05-c8d358136fe9\" (UID: \"2aa07e4b-bd11-4810-af05-c8d358136fe9\") " Oct 30 23:39:47.357284 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.357210 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-xks-ca\") pod \"2aa07e4b-bd11-4810-af05-c8d358136fe9\" (UID: \"2aa07e4b-bd11-4810-af05-c8d358136fe9\") " Oct 30 23:39:47.357284 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.357248 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa07e4b-bd11-4810-af05-c8d358136fe9-kube-api-access-q4rhg\" (UniqueName: \"kubernetes.io/projected/2aa07e4b-bd11-4810-af05-c8d358136fe9-kube-api-access-q4rhg\") pod \"2aa07e4b-bd11-4810-af05-c8d358136fe9\" (UID: \"2aa07e4b-bd11-4810-af05-c8d358136fe9\") " Oct 30 23:39:47.357507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.357454 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-openshift-ca" pod "2aa07e4b-bd11-4810-af05-c8d358136fe9" (UID: "2aa07e4b-bd11-4810-af05-c8d358136fe9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:39:47.357922 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.357903 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:47.359249 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.359228 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa07e4b-bd11-4810-af05-c8d358136fe9-kube-api-access-q4rhg" pod "2aa07e4b-bd11-4810-af05-c8d358136fe9" (UID: "2aa07e4b-bd11-4810-af05-c8d358136fe9"). InnerVolumeSpecName "kube-api-access-q4rhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:39:47.359323 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.359267 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-xks-ca" pod "2aa07e4b-bd11-4810-af05-c8d358136fe9" (UID: "2aa07e4b-bd11-4810-af05-c8d358136fe9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:39:47.459279 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.459251 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa07e4b-bd11-4810-af05-c8d358136fe9-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:47.459279 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:47.459280 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4rhg\" (UniqueName: \"kubernetes.io/projected/2aa07e4b-bd11-4810-af05-c8d358136fe9-kube-api-access-q4rhg\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:48.104842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:48.104814 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" Oct 30 23:39:48.105249 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:48.104848 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc" event={"ID":"2aa07e4b-bd11-4810-af05-c8d358136fe9","Type":"ContainerDied","Data":"15b66026f44c83a6dbad643139d3f3a32cb95550f725c59aef05817f45f33134"} Oct 30 23:39:48.105249 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:48.104896 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15b66026f44c83a6dbad643139d3f3a32cb95550f725c59aef05817f45f33134" Oct 30 23:39:48.106657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:48.106633 7570 generic.go:356] "Generic (PLEG): container finished" podID="0a2cbe38-7fc7-4274-8834-375b73af158a" containerID="a3349f1ce336672604ffe9559c69a2151851d01ef5f285d5a2a6a0ae543db807" exitCode=0 Oct 30 23:39:48.106790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:48.106707 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" event={"ID":"0a2cbe38-7fc7-4274-8834-375b73af158a","Type":"ContainerDied","Data":"a3349f1ce336672604ffe9559c69a2151851d01ef5f285d5a2a6a0ae543db807"} Oct 30 23:39:49.298808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:49.298780 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:49.388288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:49.388207 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-openshift-ca\") pod \"0a2cbe38-7fc7-4274-8834-375b73af158a\" (UID: \"0a2cbe38-7fc7-4274-8834-375b73af158a\") " Oct 30 23:39:49.388437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:49.388300 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2cbe38-7fc7-4274-8834-375b73af158a-kube-api-access-j9wqr\" (UniqueName: \"kubernetes.io/projected/0a2cbe38-7fc7-4274-8834-375b73af158a-kube-api-access-j9wqr\") pod \"0a2cbe38-7fc7-4274-8834-375b73af158a\" (UID: \"0a2cbe38-7fc7-4274-8834-375b73af158a\") " Oct 30 23:39:49.388437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:49.388376 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-xks-ca\") pod \"0a2cbe38-7fc7-4274-8834-375b73af158a\" (UID: \"0a2cbe38-7fc7-4274-8834-375b73af158a\") " Oct 30 23:39:49.388598 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:49.388546 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-openshift-ca" pod "0a2cbe38-7fc7-4274-8834-375b73af158a" (UID: "0a2cbe38-7fc7-4274-8834-375b73af158a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:39:49.388926 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:49.388903 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:49.390526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:49.390497 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2cbe38-7fc7-4274-8834-375b73af158a-kube-api-access-j9wqr" pod "0a2cbe38-7fc7-4274-8834-375b73af158a" (UID: "0a2cbe38-7fc7-4274-8834-375b73af158a"). InnerVolumeSpecName "kube-api-access-j9wqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:39:49.390611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:49.390527 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-xks-ca" pod "0a2cbe38-7fc7-4274-8834-375b73af158a" (UID: "0a2cbe38-7fc7-4274-8834-375b73af158a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:39:49.490227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:49.490200 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9wqr\" (UniqueName: \"kubernetes.io/projected/0a2cbe38-7fc7-4274-8834-375b73af158a-kube-api-access-j9wqr\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:49.490227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:49.490226 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2cbe38-7fc7-4274-8834-375b73af158a-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:50.120896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:50.120870 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" Oct 30 23:39:50.121072 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:50.120869 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q" event={"ID":"0a2cbe38-7fc7-4274-8834-375b73af158a","Type":"ContainerDied","Data":"63c8bb8b0d2286b3753f13dba2fb93cf97e772bf1ab2eed5d6b9b0edbf1fb461"} Oct 30 23:39:50.121072 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:50.120985 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63c8bb8b0d2286b3753f13dba2fb93cf97e772bf1ab2eed5d6b9b0edbf1fb461" Oct 30 23:39:55.644030 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.643998 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc"] Oct 30 23:39:55.644844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.644825 7570 cpu_manager.go:414] "containerMap: removing container" podUID="2aa07e4b-bd11-4810-af05-c8d358136fe9" containerName="revoke-keys" Oct 30 23:39:55.644950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.644846 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa07e4b-bd11-4810-af05-c8d358136fe9" containerName="revoke-keys" Oct 30 23:39:55.644950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.644879 7570 cpu_manager.go:414] "containerMap: removing container" podUID="0a2cbe38-7fc7-4274-8834-375b73af158a" containerName="revoke-keys" Oct 30 23:39:55.644950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.644886 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2cbe38-7fc7-4274-8834-375b73af158a" containerName="revoke-keys" Oct 30 23:39:55.645099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.645020 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa07e4b-bd11-4810-af05-c8d358136fe9" containerName="revoke-keys" Oct 30 23:39:55.645099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.645039 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2cbe38-7fc7-4274-8834-375b73af158a" containerName="revoke-keys" Oct 30 23:39:55.649000 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.648978 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:39:55.654991 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.654966 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:39:55.655122 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.655011 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:39:55.657733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.657712 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc"] Oct 30 23:39:55.759237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.759209 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prf42\" (UniqueName: \"kubernetes.io/projected/ac0ce57c-9e80-4409-9957-957d5efcddff-kube-api-access-prf42\") pod \"maas-api-revoke-sub-79f8ebb2-rtphc\" (UID: \"ac0ce57c-9e80-4409-9957-957d5efcddff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:39:55.759386 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.759244 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f8ebb2-rtphc\" (UID: \"ac0ce57c-9e80-4409-9957-957d5efcddff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:39:55.759429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.759390 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f8ebb2-rtphc\" (UID: \"ac0ce57c-9e80-4409-9957-957d5efcddff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:39:55.860534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.860501 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prf42\" (UniqueName: \"kubernetes.io/projected/ac0ce57c-9e80-4409-9957-957d5efcddff-kube-api-access-prf42\") pod \"maas-api-revoke-sub-79f8ebb2-rtphc\" (UID: \"ac0ce57c-9e80-4409-9957-957d5efcddff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:39:55.860703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.860547 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f8ebb2-rtphc\" (UID: \"ac0ce57c-9e80-4409-9957-957d5efcddff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:39:55.860703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.860639 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f8ebb2-rtphc\" (UID: \"ac0ce57c-9e80-4409-9957-957d5efcddff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:39:55.861239 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.861204 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f8ebb2-rtphc\" (UID: \"ac0ce57c-9e80-4409-9957-957d5efcddff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:39:55.862979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.862960 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f8ebb2-rtphc\" (UID: \"ac0ce57c-9e80-4409-9957-957d5efcddff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:39:55.879103 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.879080 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prf42\" (UniqueName: \"kubernetes.io/projected/ac0ce57c-9e80-4409-9957-957d5efcddff-kube-api-access-prf42\") pod \"maas-api-revoke-sub-79f8ebb2-rtphc\" (UID: \"ac0ce57c-9e80-4409-9957-957d5efcddff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:39:55.965417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:55.965338 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:39:56.252722 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.252693 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc"] Oct 30 23:39:56.253636 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:39:56.253608 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0ce57c_9e80_4409_9957_957d5efcddff.slice/crio-c41bc9890cb9af57dd44e6c672336e73997bfefee8fee128e3dd3bdd9f317b0f WatchSource:0}: Error finding container c41bc9890cb9af57dd44e6c672336e73997bfefee8fee128e3dd3bdd9f317b0f: Status 404 returned error can't find the container with id c41bc9890cb9af57dd44e6c672336e73997bfefee8fee128e3dd3bdd9f317b0f Oct 30 23:39:56.777336 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.777308 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92"] Oct 30 23:39:56.781356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.781337 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:39:56.789967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.789947 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92"] Oct 30 23:39:56.872622 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.872542 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbd78f4b-6wx92\" (UID: \"0c202e0d-cc38-45d4-ad64-b953c14160ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:39:56.872781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.872646 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svxrh\" (UniqueName: \"kubernetes.io/projected/0c202e0d-cc38-45d4-ad64-b953c14160ad-kube-api-access-svxrh\") pod \"maas-api-revoke-sub-bbd78f4b-6wx92\" (UID: \"0c202e0d-cc38-45d4-ad64-b953c14160ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:39:56.872842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.872817 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbd78f4b-6wx92\" (UID: \"0c202e0d-cc38-45d4-ad64-b953c14160ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:39:56.974363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.974328 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbd78f4b-6wx92\" (UID: \"0c202e0d-cc38-45d4-ad64-b953c14160ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:39:56.974564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.974429 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbd78f4b-6wx92\" (UID: \"0c202e0d-cc38-45d4-ad64-b953c14160ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:39:56.974564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.974482 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svxrh\" (UniqueName: \"kubernetes.io/projected/0c202e0d-cc38-45d4-ad64-b953c14160ad-kube-api-access-svxrh\") pod \"maas-api-revoke-sub-bbd78f4b-6wx92\" (UID: \"0c202e0d-cc38-45d4-ad64-b953c14160ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:39:56.974962 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.974943 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbd78f4b-6wx92\" (UID: \"0c202e0d-cc38-45d4-ad64-b953c14160ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:39:56.977020 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.976994 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbd78f4b-6wx92\" (UID: \"0c202e0d-cc38-45d4-ad64-b953c14160ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:39:56.986338 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:56.986316 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svxrh\" (UniqueName: \"kubernetes.io/projected/0c202e0d-cc38-45d4-ad64-b953c14160ad-kube-api-access-svxrh\") pod \"maas-api-revoke-sub-bbd78f4b-6wx92\" (UID: \"0c202e0d-cc38-45d4-ad64-b953c14160ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:39:57.097929 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:57.097902 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:39:57.179069 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:57.179039 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" event={"ID":"ac0ce57c-9e80-4409-9957-957d5efcddff","Type":"ContainerStarted","Data":"8bd49990dcb203d479e1d86ce65d3b28b1b7edd83ef9d1d0fe8d17d0cb4c1ace"} Oct 30 23:39:57.179224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:57.179101 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" event={"ID":"ac0ce57c-9e80-4409-9957-957d5efcddff","Type":"ContainerStarted","Data":"c41bc9890cb9af57dd44e6c672336e73997bfefee8fee128e3dd3bdd9f317b0f"} Oct 30 23:39:57.198151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:57.198094 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" podStartSLOduration=2.198075037 podStartE2EDuration="2.198075037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:57.197347481 +0000 UTC m=+1208.716348572" watchObservedRunningTime="2026-10-30 23:39:57.198075037 +0000 UTC m=+1208.717076127" Oct 30 23:39:57.383261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:57.383239 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92"] Oct 30 23:39:57.384616 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:39:57.384587 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c202e0d_cc38_45d4_ad64_b953c14160ad.slice/crio-c86efb511c51bea218924be4b830fb351271d4c4a99b46a61b88ea7795f84b61 WatchSource:0}: Error finding container c86efb511c51bea218924be4b830fb351271d4c4a99b46a61b88ea7795f84b61: Status 404 returned error can't find the container with id c86efb511c51bea218924be4b830fb351271d4c4a99b46a61b88ea7795f84b61 Oct 30 23:39:58.186609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:58.186572 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" event={"ID":"0c202e0d-cc38-45d4-ad64-b953c14160ad","Type":"ContainerStarted","Data":"4b95e5f209f3cbf2a416c6d7059d556e05005bbf01796d10a7d0b5652c673a34"} Oct 30 23:39:58.187115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:58.186674 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" event={"ID":"0c202e0d-cc38-45d4-ad64-b953c14160ad","Type":"ContainerStarted","Data":"c86efb511c51bea218924be4b830fb351271d4c4a99b46a61b88ea7795f84b61"} Oct 30 23:39:58.188349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:58.188325 7570 generic.go:356] "Generic (PLEG): container finished" podID="ac0ce57c-9e80-4409-9957-957d5efcddff" containerID="8bd49990dcb203d479e1d86ce65d3b28b1b7edd83ef9d1d0fe8d17d0cb4c1ace" exitCode=0 Oct 30 23:39:58.188501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:58.188405 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" event={"ID":"ac0ce57c-9e80-4409-9957-957d5efcddff","Type":"ContainerDied","Data":"8bd49990dcb203d479e1d86ce65d3b28b1b7edd83ef9d1d0fe8d17d0cb4c1ace"} Oct 30 23:39:58.206139 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:58.206095 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" podStartSLOduration=2.206084549 podStartE2EDuration="2.206084549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:58.204501164 +0000 UTC m=+1209.723502254" watchObservedRunningTime="2026-10-30 23:39:58.206084549 +0000 UTC m=+1209.725085683" Oct 30 23:39:59.195617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.195587 7570 generic.go:356] "Generic (PLEG): container finished" podID="0c202e0d-cc38-45d4-ad64-b953c14160ad" containerID="4b95e5f209f3cbf2a416c6d7059d556e05005bbf01796d10a7d0b5652c673a34" exitCode=0 Oct 30 23:39:59.196044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.195713 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" event={"ID":"0c202e0d-cc38-45d4-ad64-b953c14160ad","Type":"ContainerDied","Data":"4b95e5f209f3cbf2a416c6d7059d556e05005bbf01796d10a7d0b5652c673a34"} Oct 30 23:39:59.327236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.327202 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn"] Oct 30 23:39:59.338575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.338546 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:39:59.361577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.347263 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn"] Oct 30 23:39:59.383345 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.383198 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:39:59.406454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.406422 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj8js\" (UniqueName: \"kubernetes.io/projected/723bafe3-3781-4262-a5ea-ef0669d065d4-kube-api-access-qj8js\") pod \"maas-api-revoke-sub-5ee37093-f55nn\" (UID: \"723bafe3-3781-4262-a5ea-ef0669d065d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:39:59.406685 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.406662 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ee37093-f55nn\" (UID: \"723bafe3-3781-4262-a5ea-ef0669d065d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:39:59.406751 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.406707 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ee37093-f55nn\" (UID: \"723bafe3-3781-4262-a5ea-ef0669d065d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:39:59.428174 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:39:59.428137 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:39:59.507828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.507757 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-openshift-ca\") pod \"ac0ce57c-9e80-4409-9957-957d5efcddff\" (UID: \"ac0ce57c-9e80-4409-9957-957d5efcddff\") " Oct 30 23:39:59.507989 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.507842 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0ce57c-9e80-4409-9957-957d5efcddff-kube-api-access-prf42\" (UniqueName: \"kubernetes.io/projected/ac0ce57c-9e80-4409-9957-957d5efcddff-kube-api-access-prf42\") pod \"ac0ce57c-9e80-4409-9957-957d5efcddff\" (UID: \"ac0ce57c-9e80-4409-9957-957d5efcddff\") " Oct 30 23:39:59.507989 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.507940 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-xks-ca\") pod \"ac0ce57c-9e80-4409-9957-957d5efcddff\" (UID: \"ac0ce57c-9e80-4409-9957-957d5efcddff\") " Oct 30 23:39:59.508179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.508155 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-openshift-ca" pod "ac0ce57c-9e80-4409-9957-957d5efcddff" (UID: "ac0ce57c-9e80-4409-9957-957d5efcddff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:39:59.508312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.508295 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj8js\" (UniqueName: \"kubernetes.io/projected/723bafe3-3781-4262-a5ea-ef0669d065d4-kube-api-access-qj8js\") pod \"maas-api-revoke-sub-5ee37093-f55nn\" (UID: \"723bafe3-3781-4262-a5ea-ef0669d065d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:39:59.508489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.508433 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ee37093-f55nn\" (UID: \"723bafe3-3781-4262-a5ea-ef0669d065d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:39:59.508621 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.508508 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ee37093-f55nn\" (UID: \"723bafe3-3781-4262-a5ea-ef0669d065d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:39:59.508621 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.508572 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:59.509127 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.509105 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ee37093-f55nn\" (UID: \"723bafe3-3781-4262-a5ea-ef0669d065d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:39:59.510036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.510015 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-xks-ca" pod "ac0ce57c-9e80-4409-9957-957d5efcddff" (UID: "ac0ce57c-9e80-4409-9957-957d5efcddff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:39:59.510121 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.510102 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0ce57c-9e80-4409-9957-957d5efcddff-kube-api-access-prf42" pod "ac0ce57c-9e80-4409-9957-957d5efcddff" (UID: "ac0ce57c-9e80-4409-9957-957d5efcddff"). InnerVolumeSpecName "kube-api-access-prf42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:39:59.511024 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.510999 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ee37093-f55nn\" (UID: \"723bafe3-3781-4262-a5ea-ef0669d065d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:39:59.520843 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.520824 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj8js\" (UniqueName: \"kubernetes.io/projected/723bafe3-3781-4262-a5ea-ef0669d065d4-kube-api-access-qj8js\") pod \"maas-api-revoke-sub-5ee37093-f55nn\" (UID: \"723bafe3-3781-4262-a5ea-ef0669d065d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:39:59.610293 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.610269 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prf42\" (UniqueName: \"kubernetes.io/projected/ac0ce57c-9e80-4409-9957-957d5efcddff-kube-api-access-prf42\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:59.610293 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.610293 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0ce57c-9e80-4409-9957-957d5efcddff-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:39:59.703124 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:39:59.703073 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:40:00.047966 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.047722 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn"] Oct 30 23:40:00.280488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.279987 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" event={"ID":"ac0ce57c-9e80-4409-9957-957d5efcddff","Type":"ContainerDied","Data":"c41bc9890cb9af57dd44e6c672336e73997bfefee8fee128e3dd3bdd9f317b0f"} Oct 30 23:40:00.280488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.280052 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c41bc9890cb9af57dd44e6c672336e73997bfefee8fee128e3dd3bdd9f317b0f" Oct 30 23:40:00.280488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.280133 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc" Oct 30 23:40:00.326538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.326160 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" event={"ID":"723bafe3-3781-4262-a5ea-ef0669d065d4","Type":"ContainerStarted","Data":"de2c6d43723e074ece79331ae3e3a89ed5095407f55a72f40d4a6d87b88d50ed"} Oct 30 23:40:00.659635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.659604 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:40:00.751494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.749847 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-openshift-ca\") pod \"0c202e0d-cc38-45d4-ad64-b953c14160ad\" (UID: \"0c202e0d-cc38-45d4-ad64-b953c14160ad\") " Oct 30 23:40:00.751494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.750076 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-xks-ca\") pod \"0c202e0d-cc38-45d4-ad64-b953c14160ad\" (UID: \"0c202e0d-cc38-45d4-ad64-b953c14160ad\") " Oct 30 23:40:00.751494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.750154 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c202e0d-cc38-45d4-ad64-b953c14160ad-kube-api-access-svxrh\" (UniqueName: \"kubernetes.io/projected/0c202e0d-cc38-45d4-ad64-b953c14160ad-kube-api-access-svxrh\") pod \"0c202e0d-cc38-45d4-ad64-b953c14160ad\" (UID: \"0c202e0d-cc38-45d4-ad64-b953c14160ad\") " Oct 30 23:40:00.773514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.767881 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-openshift-ca" pod "0c202e0d-cc38-45d4-ad64-b953c14160ad" (UID: "0c202e0d-cc38-45d4-ad64-b953c14160ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:40:00.773514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.771799 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c202e0d-cc38-45d4-ad64-b953c14160ad-kube-api-access-svxrh" pod "0c202e0d-cc38-45d4-ad64-b953c14160ad" (UID: "0c202e0d-cc38-45d4-ad64-b953c14160ad"). InnerVolumeSpecName "kube-api-access-svxrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:00.773787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.773770 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-xks-ca" pod "0c202e0d-cc38-45d4-ad64-b953c14160ad" (UID: "0c202e0d-cc38-45d4-ad64-b953c14160ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:40:00.852502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.851772 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:00.852502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.851817 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c202e0d-cc38-45d4-ad64-b953c14160ad-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:00.852502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:00.851835 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svxrh\" (UniqueName: \"kubernetes.io/projected/0c202e0d-cc38-45d4-ad64-b953c14160ad-kube-api-access-svxrh\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:01.334801 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:01.334762 7570 generic.go:356] "Generic (PLEG): container finished" podID="723bafe3-3781-4262-a5ea-ef0669d065d4" containerID="63002016fc0ff19546e215b9c75e58734088649be676d5e13403a6adb62f5579" exitCode=0 Oct 30 23:40:01.335213 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:01.334811 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" event={"ID":"723bafe3-3781-4262-a5ea-ef0669d065d4","Type":"ContainerDied","Data":"63002016fc0ff19546e215b9c75e58734088649be676d5e13403a6adb62f5579"} Oct 30 23:40:01.336436 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:01.336415 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" Oct 30 23:40:01.336587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:01.336435 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92" event={"ID":"0c202e0d-cc38-45d4-ad64-b953c14160ad","Type":"ContainerDied","Data":"c86efb511c51bea218924be4b830fb351271d4c4a99b46a61b88ea7795f84b61"} Oct 30 23:40:01.336587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:01.336487 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c86efb511c51bea218924be4b830fb351271d4c4a99b46a61b88ea7795f84b61" Oct 30 23:40:02.507408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:02.507379 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:40:02.571711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:02.571685 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-openshift-ca\") pod \"723bafe3-3781-4262-a5ea-ef0669d065d4\" (UID: \"723bafe3-3781-4262-a5ea-ef0669d065d4\") " Oct 30 23:40:02.571892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:02.571796 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-xks-ca\") pod \"723bafe3-3781-4262-a5ea-ef0669d065d4\" (UID: \"723bafe3-3781-4262-a5ea-ef0669d065d4\") " Oct 30 23:40:02.572014 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:02.571998 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723bafe3-3781-4262-a5ea-ef0669d065d4-kube-api-access-qj8js\" (UniqueName: \"kubernetes.io/projected/723bafe3-3781-4262-a5ea-ef0669d065d4-kube-api-access-qj8js\") pod \"723bafe3-3781-4262-a5ea-ef0669d065d4\" (UID: \"723bafe3-3781-4262-a5ea-ef0669d065d4\") " Oct 30 23:40:02.572111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:02.572083 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-openshift-ca" pod "723bafe3-3781-4262-a5ea-ef0669d065d4" (UID: "723bafe3-3781-4262-a5ea-ef0669d065d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:40:02.572741 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:02.572716 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:02.574167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:02.574136 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-xks-ca" pod "723bafe3-3781-4262-a5ea-ef0669d065d4" (UID: "723bafe3-3781-4262-a5ea-ef0669d065d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:40:02.574259 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:02.574188 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723bafe3-3781-4262-a5ea-ef0669d065d4-kube-api-access-qj8js" pod "723bafe3-3781-4262-a5ea-ef0669d065d4" (UID: "723bafe3-3781-4262-a5ea-ef0669d065d4"). InnerVolumeSpecName "kube-api-access-qj8js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:02.674409 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:02.674377 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj8js\" (UniqueName: \"kubernetes.io/projected/723bafe3-3781-4262-a5ea-ef0669d065d4-kube-api-access-qj8js\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:02.674554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:02.674412 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723bafe3-3781-4262-a5ea-ef0669d065d4-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:03.349730 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:03.349706 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" Oct 30 23:40:03.349913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:03.349742 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn" event={"ID":"723bafe3-3781-4262-a5ea-ef0669d065d4","Type":"ContainerDied","Data":"de2c6d43723e074ece79331ae3e3a89ed5095407f55a72f40d4a6d87b88d50ed"} Oct 30 23:40:03.349913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:03.349789 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de2c6d43723e074ece79331ae3e3a89ed5095407f55a72f40d4a6d87b88d50ed" Oct 30 23:40:09.327995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.327958 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb"] Oct 30 23:40:09.328515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.328502 7570 cpu_manager.go:414] "containerMap: removing container" podUID="0c202e0d-cc38-45d4-ad64-b953c14160ad" containerName="revoke-keys" Oct 30 23:40:09.328577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.328516 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c202e0d-cc38-45d4-ad64-b953c14160ad" containerName="revoke-keys" Oct 30 23:40:09.328577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.328531 7570 cpu_manager.go:414] "containerMap: removing container" podUID="723bafe3-3781-4262-a5ea-ef0669d065d4" containerName="revoke-keys" Oct 30 23:40:09.328577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.328535 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723bafe3-3781-4262-a5ea-ef0669d065d4" containerName="revoke-keys" Oct 30 23:40:09.328577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.328567 7570 cpu_manager.go:414] "containerMap: removing container" podUID="ac0ce57c-9e80-4409-9957-957d5efcddff" containerName="revoke-keys" Oct 30 23:40:09.328577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.328572 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0ce57c-9e80-4409-9957-957d5efcddff" containerName="revoke-keys" Oct 30 23:40:09.328724 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.328672 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c202e0d-cc38-45d4-ad64-b953c14160ad" containerName="revoke-keys" Oct 30 23:40:09.328724 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.328681 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="723bafe3-3781-4262-a5ea-ef0669d065d4" containerName="revoke-keys" Oct 30 23:40:09.328724 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.328691 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0ce57c-9e80-4409-9957-957d5efcddff" containerName="revoke-keys" Oct 30 23:40:09.332591 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.332567 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:09.338801 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.338659 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:40:09.355845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.355797 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:40:09.362719 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.362285 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb"] Oct 30 23:40:09.468681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.468651 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84a9e680-v2kdb\" (UID: \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:09.468844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.468810 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84a9e680-v2kdb\" (UID: \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:09.468901 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.468863 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gclnb\" (UniqueName: \"kubernetes.io/projected/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-kube-api-access-gclnb\") pod \"maas-api-revoke-sub-84a9e680-v2kdb\" (UID: \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:09.569890 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.569862 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84a9e680-v2kdb\" (UID: \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:09.570047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.569970 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84a9e680-v2kdb\" (UID: \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:09.570047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.570008 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gclnb\" (UniqueName: \"kubernetes.io/projected/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-kube-api-access-gclnb\") pod \"maas-api-revoke-sub-84a9e680-v2kdb\" (UID: \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:09.577520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.571069 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84a9e680-v2kdb\" (UID: \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:09.577520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.573573 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84a9e680-v2kdb\" (UID: \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:09.583247 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.583187 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gclnb\" (UniqueName: \"kubernetes.io/projected/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-kube-api-access-gclnb\") pod \"maas-api-revoke-sub-84a9e680-v2kdb\" (UID: \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:09.673890 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.673867 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:09.952587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:09.952555 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb"] Oct 30 23:40:09.954147 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:40:09.954109 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2b2e0d_f349_455b_91f3_5186f9dfddeb.slice/crio-703834dccb45f0a8460d7ca93e79c74c3b533e63e5d348fb9fffe534102d9b85 WatchSource:0}: Error finding container 703834dccb45f0a8460d7ca93e79c74c3b533e63e5d348fb9fffe534102d9b85: Status 404 returned error can't find the container with id 703834dccb45f0a8460d7ca93e79c74c3b533e63e5d348fb9fffe534102d9b85 Oct 30 23:40:10.397647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:10.397619 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" event={"ID":"fc2b2e0d-f349-455b-91f3-5186f9dfddeb","Type":"ContainerStarted","Data":"703834dccb45f0a8460d7ca93e79c74c3b533e63e5d348fb9fffe534102d9b85"} Oct 30 23:40:10.938709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:10.938667 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl"] Oct 30 23:40:10.943073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:10.943054 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:10.956135 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:10.956109 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl"] Oct 30 23:40:10.995353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:10.995325 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b"] Oct 30 23:40:10.999519 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:10.999501 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.009063 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.009041 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d3dcca-openshift-default-dockercfg-mtkk6\"" Oct 30 23:40:11.012200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.012180 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b"] Oct 30 23:40:11.089901 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.089873 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.090023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.089909 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.090023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.089927 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.090023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090011 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.090127 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090064 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.090127 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090107 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5qh\" (UniqueName: \"kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-kube-api-access-wb5qh\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.090189 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090170 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792ae9e3-7af7-4767-900f-409a34cc02ac-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.090222 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090216 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.090255 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090233 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.090336 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090323 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.090382 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090364 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.090501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090484 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf4s2\" (UniqueName: \"kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-kube-api-access-sf4s2\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.090570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090558 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.090612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090595 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.090662 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090614 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.090662 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090642 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.090820 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090758 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.090820 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.090804 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.192755 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.192660 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.192958 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.192765 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.192958 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.192843 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4s2\" (UniqueName: \"kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-kube-api-access-sf4s2\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.192958 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.192903 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.192958 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.192940 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.193184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.192963 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.193184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.192988 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.193184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193013 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.193184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193035 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.193184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193064 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.193184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193082 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.193184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193096 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.193184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193117 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.193184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193148 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.193184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193175 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5qh\" (UniqueName: \"kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-kube-api-access-wb5qh\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193238 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792ae9e3-7af7-4767-900f-409a34cc02ac-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193296 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193320 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193540 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193777 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.193817 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.194167 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.194731 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792ae9e3-7af7-4767-900f-409a34cc02ac-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.194843 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.195322 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.195877 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.196334 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.196660 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.196983 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.198916 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.199763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.199248 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.200797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.200772 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.206387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.206361 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.206387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.206373 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4s2\" (UniqueName: \"kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-kube-api-access-sf4s2\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.206620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.206431 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.206620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.206436 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5qh\" (UniqueName: \"kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-kube-api-access-wb5qh\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.260491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.260434 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:11.376418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.376374 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:11.406821 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.406786 7570 generic.go:356] "Generic (PLEG): container finished" podID="fc2b2e0d-f349-455b-91f3-5186f9dfddeb" containerID="4ae427a6d9f9ea44211f4ddbb96cc383f19e4cc46d8c01b8dc3fc8ba35296f0e" exitCode=0 Oct 30 23:40:11.407170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.406911 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" event={"ID":"fc2b2e0d-f349-455b-91f3-5186f9dfddeb","Type":"ContainerDied","Data":"4ae427a6d9f9ea44211f4ddbb96cc383f19e4cc46d8c01b8dc3fc8ba35296f0e"} Oct 30 23:40:11.545521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.545497 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl"] Oct 30 23:40:11.546235 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:40:11.546210 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792ae9e3_7af7_4767_900f_409a34cc02ac.slice/crio-979cb4ba7f8ec82c00e285c434b8382fc1047129086878ceaa32ab683ec6fd07 WatchSource:0}: Error finding container 979cb4ba7f8ec82c00e285c434b8382fc1047129086878ceaa32ab683ec6fd07: Status 404 returned error can't find the container with id 979cb4ba7f8ec82c00e285c434b8382fc1047129086878ceaa32ab683ec6fd07 Oct 30 23:40:11.548768 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.548735 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:11.548857 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.548799 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:11.548857 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.548827 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:11.664418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.664394 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b"] Oct 30 23:40:11.666810 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:40:11.666782 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdee70e9_f8d5_40e2_a4e4_bb2fc009db4f.slice/crio-5cd54dc16767b953c9edebbddcc553b00da872d7ec853df4ed83626ecfd5335f WatchSource:0}: Error finding container 5cd54dc16767b953c9edebbddcc553b00da872d7ec853df4ed83626ecfd5335f: Status 404 returned error can't find the container with id 5cd54dc16767b953c9edebbddcc553b00da872d7ec853df4ed83626ecfd5335f Oct 30 23:40:11.670304 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.670266 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:11.670416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.670371 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:11.670486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:11.670450 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:12.414747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.414712 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" event={"ID":"792ae9e3-7af7-4767-900f-409a34cc02ac","Type":"ContainerStarted","Data":"7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f"} Oct 30 23:40:12.415162 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.414768 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" event={"ID":"792ae9e3-7af7-4767-900f-409a34cc02ac","Type":"ContainerStarted","Data":"979cb4ba7f8ec82c00e285c434b8382fc1047129086878ceaa32ab683ec6fd07"} Oct 30 23:40:12.416442 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.416418 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" event={"ID":"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f","Type":"ContainerStarted","Data":"29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b"} Oct 30 23:40:12.416607 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.416473 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" event={"ID":"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f","Type":"ContainerStarted","Data":"5cd54dc16767b953c9edebbddcc553b00da872d7ec853df4ed83626ecfd5335f"} Oct 30 23:40:12.440943 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.440900 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" podStartSLOduration=2.440886656 podStartE2EDuration="2.440886656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:12.438902041 +0000 UTC m=+1223.957903128" watchObservedRunningTime="2026-10-30 23:40:12.440886656 +0000 UTC m=+1223.959887824" Oct 30 23:40:12.465202 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.465151 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podStartSLOduration=2.465138561 podStartE2EDuration="2.465138561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:12.4618057 +0000 UTC m=+1223.980806791" watchObservedRunningTime="2026-10-30 23:40:12.465138561 +0000 UTC m=+1223.984139701" Oct 30 23:40:12.659834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.659801 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:12.829911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.829875 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-kube-api-access-gclnb\" (UniqueName: \"kubernetes.io/projected/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-kube-api-access-gclnb\") pod \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\" (UID: \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\") " Oct 30 23:40:12.839486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.830552 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-openshift-ca\") pod \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\" (UID: \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\") " Oct 30 23:40:12.839846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.839819 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-xks-ca\") pod \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\" (UID: \"fc2b2e0d-f349-455b-91f3-5186f9dfddeb\") " Oct 30 23:40:12.840873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.834819 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-openshift-ca" pod "fc2b2e0d-f349-455b-91f3-5186f9dfddeb" (UID: "fc2b2e0d-f349-455b-91f3-5186f9dfddeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:40:12.841051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.841023 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-kube-api-access-gclnb" pod "fc2b2e0d-f349-455b-91f3-5186f9dfddeb" (UID: "fc2b2e0d-f349-455b-91f3-5186f9dfddeb"). InnerVolumeSpecName "kube-api-access-gclnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:12.845524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.843859 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-xks-ca" pod "fc2b2e0d-f349-455b-91f3-5186f9dfddeb" (UID: "fc2b2e0d-f349-455b-91f3-5186f9dfddeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:40:12.942028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.941995 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gclnb\" (UniqueName: \"kubernetes.io/projected/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-kube-api-access-gclnb\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:12.942028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.942026 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:12.942028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:12.942038 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2b2e0d-f349-455b-91f3-5186f9dfddeb-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:13.260923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:13.260839 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:13.261990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:13.261963 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 23:40:13.262143 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:13.262019 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" podUID="792ae9e3-7af7-4767-900f-409a34cc02ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 23:40:13.376999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:13.376966 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:13.378519 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:13.378494 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 23:40:13.378676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:13.378547 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 23:40:13.446760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:13.446725 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" Oct 30 23:40:13.446760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:13.446746 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb" event={"ID":"fc2b2e0d-f349-455b-91f3-5186f9dfddeb","Type":"ContainerDied","Data":"703834dccb45f0a8460d7ca93e79c74c3b533e63e5d348fb9fffe534102d9b85"} Oct 30 23:40:13.447189 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:13.446806 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="703834dccb45f0a8460d7ca93e79c74c3b533e63e5d348fb9fffe534102d9b85" Oct 30 23:40:14.260908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:14.260862 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 23:40:14.261079 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:14.260923 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" podUID="792ae9e3-7af7-4767-900f-409a34cc02ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 23:40:14.377533 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:14.377497 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 23:40:14.377711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:14.377563 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 23:40:15.261980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:15.261942 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 23:40:15.262387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:15.262001 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" podUID="792ae9e3-7af7-4767-900f-409a34cc02ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 23:40:15.377630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:15.377596 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 23:40:15.377826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:15.377658 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 23:40:16.262568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:16.262532 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 23:40:16.262962 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:16.262603 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" podUID="792ae9e3-7af7-4767-900f-409a34cc02ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 23:40:16.379487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:16.378994 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 23:40:16.379487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:16.379077 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 23:40:17.267486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:17.261849 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 23:40:17.267486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:17.261916 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" podUID="792ae9e3-7af7-4767-900f-409a34cc02ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 23:40:17.382487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:17.381964 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 23:40:17.382487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:17.382026 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 23:40:17.531546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:17.531276 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:40:17.649492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:17.647571 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:40:17.847485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:17.842570 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:40:17.981944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:17.980542 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:40:18.064489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:18.063576 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:40:18.265551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:18.263842 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 23:40:18.265551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:18.263904 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" podUID="792ae9e3-7af7-4767-900f-409a34cc02ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 23:40:18.290487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:18.289930 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:40:18.335487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:18.319411 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:40:18.378805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:18.378776 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 23:40:18.379621 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:18.379050 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 23:40:19.261242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.261206 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 23:40:19.261477 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.261285 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" podUID="792ae9e3-7af7-4767-900f-409a34cc02ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 23:40:19.379939 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.379910 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 23:40:19.380494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.380430 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 23:40:19.586819 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.586786 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4"] Oct 30 23:40:19.588314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.588290 7570 cpu_manager.go:414] "containerMap: removing container" podUID="fc2b2e0d-f349-455b-91f3-5186f9dfddeb" containerName="revoke-keys" Oct 30 23:40:19.588480 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.588449 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc2b2e0d-f349-455b-91f3-5186f9dfddeb" containerName="revoke-keys" Oct 30 23:40:19.588979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.588785 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc2b2e0d-f349-455b-91f3-5186f9dfddeb" containerName="revoke-keys" Oct 30 23:40:19.594573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.594540 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:19.606137 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.605971 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:40:19.607164 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.607145 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:40:19.608487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.608445 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4"] Oct 30 23:40:19.639246 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.637504 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a268da2b-kgwf4\" (UID: \"e473a80f-5ef4-459c-9d88-db39f2393b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:19.639246 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.637595 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqr2\" (UniqueName: \"kubernetes.io/projected/e473a80f-5ef4-459c-9d88-db39f2393b04-kube-api-access-zvqr2\") pod \"maas-api-revoke-sub-a268da2b-kgwf4\" (UID: \"e473a80f-5ef4-459c-9d88-db39f2393b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:19.639246 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.637642 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a268da2b-kgwf4\" (UID: \"e473a80f-5ef4-459c-9d88-db39f2393b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:19.742078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.742042 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a268da2b-kgwf4\" (UID: \"e473a80f-5ef4-459c-9d88-db39f2393b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:19.742673 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.742647 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a268da2b-kgwf4\" (UID: \"e473a80f-5ef4-459c-9d88-db39f2393b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:19.742873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.742841 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqr2\" (UniqueName: \"kubernetes.io/projected/e473a80f-5ef4-459c-9d88-db39f2393b04-kube-api-access-zvqr2\") pod \"maas-api-revoke-sub-a268da2b-kgwf4\" (UID: \"e473a80f-5ef4-459c-9d88-db39f2393b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:19.744137 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.744079 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a268da2b-kgwf4\" (UID: \"e473a80f-5ef4-459c-9d88-db39f2393b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:19.758315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.758289 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqr2\" (UniqueName: \"kubernetes.io/projected/e473a80f-5ef4-459c-9d88-db39f2393b04-kube-api-access-zvqr2\") pod \"maas-api-revoke-sub-a268da2b-kgwf4\" (UID: \"e473a80f-5ef4-459c-9d88-db39f2393b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:19.758964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.758914 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a268da2b-kgwf4\" (UID: \"e473a80f-5ef4-459c-9d88-db39f2393b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:19.936815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:19.936677 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:20.267062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:20.267034 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4"] Oct 30 23:40:20.267187 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:40:20.267169 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode473a80f_5ef4_459c_9d88_db39f2393b04.slice/crio-f044757c252334bd312c865aa9394133e1c57beb105d377d2e43226e76b6151d WatchSource:0}: Error finding container f044757c252334bd312c865aa9394133e1c57beb105d377d2e43226e76b6151d: Status 404 returned error can't find the container with id f044757c252334bd312c865aa9394133e1c57beb105d377d2e43226e76b6151d Oct 30 23:40:20.382201 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:20.382169 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:20.382751 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:20.382733 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:20.384282 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:20.384260 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:40:20.582128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:20.582093 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" event={"ID":"e473a80f-5ef4-459c-9d88-db39f2393b04","Type":"ContainerStarted","Data":"f044757c252334bd312c865aa9394133e1c57beb105d377d2e43226e76b6151d"} Oct 30 23:40:20.817896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:20.817866 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:20.818763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:20.818452 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:20.824109 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:20.824003 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:40:21.610810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:21.610776 7570 generic.go:356] "Generic (PLEG): container finished" podID="e473a80f-5ef4-459c-9d88-db39f2393b04" containerID="19d696a30b62f93a455339f235b5c0512a5bd9394ab6ef21cf2a4662b94f0d3d" exitCode=0 Oct 30 23:40:21.611179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:21.610914 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" event={"ID":"e473a80f-5ef4-459c-9d88-db39f2393b04","Type":"ContainerDied","Data":"19d696a30b62f93a455339f235b5c0512a5bd9394ab6ef21cf2a4662b94f0d3d"} Oct 30 23:40:22.870549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:22.862060 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:22.974047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:22.974012 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e473a80f-5ef4-459c-9d88-db39f2393b04-kube-api-access-zvqr2\" (UniqueName: \"kubernetes.io/projected/e473a80f-5ef4-459c-9d88-db39f2393b04-kube-api-access-zvqr2\") pod \"e473a80f-5ef4-459c-9d88-db39f2393b04\" (UID: \"e473a80f-5ef4-459c-9d88-db39f2393b04\") " Oct 30 23:40:22.974248 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:22.974126 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-xks-ca\") pod \"e473a80f-5ef4-459c-9d88-db39f2393b04\" (UID: \"e473a80f-5ef4-459c-9d88-db39f2393b04\") " Oct 30 23:40:22.974248 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:22.974238 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-openshift-ca\") pod \"e473a80f-5ef4-459c-9d88-db39f2393b04\" (UID: \"e473a80f-5ef4-459c-9d88-db39f2393b04\") " Oct 30 23:40:22.974888 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:22.974850 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-openshift-ca" pod "e473a80f-5ef4-459c-9d88-db39f2393b04" (UID: "e473a80f-5ef4-459c-9d88-db39f2393b04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:40:22.979958 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:22.979905 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e473a80f-5ef4-459c-9d88-db39f2393b04-kube-api-access-zvqr2" pod "e473a80f-5ef4-459c-9d88-db39f2393b04" (UID: "e473a80f-5ef4-459c-9d88-db39f2393b04"). InnerVolumeSpecName "kube-api-access-zvqr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:22.983951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:22.983910 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-xks-ca" pod "e473a80f-5ef4-459c-9d88-db39f2393b04" (UID: "e473a80f-5ef4-459c-9d88-db39f2393b04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:40:23.076888 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:23.076853 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvqr2\" (UniqueName: \"kubernetes.io/projected/e473a80f-5ef4-459c-9d88-db39f2393b04-kube-api-access-zvqr2\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:23.076888 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:23.076886 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:23.076888 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:23.076902 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e473a80f-5ef4-459c-9d88-db39f2393b04-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:23.632822 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:23.632787 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" event={"ID":"e473a80f-5ef4-459c-9d88-db39f2393b04","Type":"ContainerDied","Data":"f044757c252334bd312c865aa9394133e1c57beb105d377d2e43226e76b6151d"} Oct 30 23:40:23.633005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:23.632825 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4" Oct 30 23:40:23.633005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:23.632846 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f044757c252334bd312c865aa9394133e1c57beb105d377d2e43226e76b6151d" Oct 30 23:40:24.734056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.734016 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp"] Oct 30 23:40:24.735549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.735521 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e473a80f-5ef4-459c-9d88-db39f2393b04" containerName="revoke-keys" Oct 30 23:40:24.735549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.735550 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e473a80f-5ef4-459c-9d88-db39f2393b04" containerName="revoke-keys" Oct 30 23:40:24.735887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.735867 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e473a80f-5ef4-459c-9d88-db39f2393b04" containerName="revoke-keys" Oct 30 23:40:24.748658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.748628 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp"] Oct 30 23:40:24.748801 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.748751 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:24.753769 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.753748 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"distinct-w4-843d8c-kserve-self-signed-certs\"" Oct 30 23:40:24.930480 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.930391 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-dshm\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:24.930792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.930766 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-model-cache\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:24.931028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.931011 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80dc7b17-04ea-4d03-a843-24891d3cda07-tls-certs\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:24.931270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.931251 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-home\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:24.931475 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.931440 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-tmp-dir\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:24.931629 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:24.931609 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shxk5\" (UniqueName: \"kubernetes.io/projected/80dc7b17-04ea-4d03-a843-24891d3cda07-kube-api-access-shxk5\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.033406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.033321 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-dshm\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.033406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.033360 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-model-cache\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.033406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.033386 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80dc7b17-04ea-4d03-a843-24891d3cda07-tls-certs\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.033731 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.033446 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-home\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.033731 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.033535 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-tmp-dir\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.033731 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.033574 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shxk5\" (UniqueName: \"kubernetes.io/projected/80dc7b17-04ea-4d03-a843-24891d3cda07-kube-api-access-shxk5\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.033899 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.033783 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-model-cache\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.034041 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.034015 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-home\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.034104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.034052 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-tmp-dir\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.035771 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.035749 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-dshm\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.035928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.035908 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80dc7b17-04ea-4d03-a843-24891d3cda07-tls-certs\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.046218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.046191 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shxk5\" (UniqueName: \"kubernetes.io/projected/80dc7b17-04ea-4d03-a843-24891d3cda07-kube-api-access-shxk5\") pod \"distinct-w4-843d8c-kserve-df87d4cff-tz6fp\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.066226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.066201 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:25.402584 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.402351 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp"] Oct 30 23:40:25.651432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.651342 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" event={"ID":"80dc7b17-04ea-4d03-a843-24891d3cda07","Type":"ContainerStarted","Data":"c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146"} Oct 30 23:40:25.651432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.651394 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" event={"ID":"80dc7b17-04ea-4d03-a843-24891d3cda07","Type":"ContainerStarted","Data":"91f7af2fe72921d591025cf070d4c1bb77167c54f55300bbb3d8496c87984bfb"} Oct 30 23:40:25.674318 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.674264 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podStartSLOduration=1.67425188 podStartE2EDuration="1.67425188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:25.6722925 +0000 UTC m=+1237.191293590" watchObservedRunningTime="2026-10-30 23:40:25.67425188 +0000 UTC m=+1237.193252970" Oct 30 23:40:25.866109 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.866075 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6"] Oct 30 23:40:25.872778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.872754 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:25.890634 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:25.890610 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6"] Oct 30 23:40:26.052508 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.052412 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.052656 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.052586 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzpd4\" (UniqueName: \"kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-kube-api-access-kzpd4\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.052656 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.052630 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.052656 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.052644 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.052780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.052720 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c029726-9be9-4ac8-a268-8761a4499b87-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.052814 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.052780 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.052814 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.052802 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.052900 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.052888 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c029726-9be9-4ac8-a268-8761a4499b87-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.053007 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.052993 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.067151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.067126 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:26.154307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.154279 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c029726-9be9-4ac8-a268-8761a4499b87-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.154498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.154396 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.154498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.154448 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.154632 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.154603 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzpd4\" (UniqueName: \"kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-kube-api-access-kzpd4\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.154689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.154664 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.154743 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.154693 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.154743 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.154721 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c029726-9be9-4ac8-a268-8761a4499b87-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.154855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.154768 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.154855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.154793 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.154855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.154839 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.154995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.154916 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.155093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.155058 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.155263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.155245 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.155510 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.155494 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c029726-9be9-4ac8-a268-8761a4499b87-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.157127 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.157107 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c029726-9be9-4ac8-a268-8761a4499b87-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.157391 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.157372 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.174763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.174730 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzpd4\" (UniqueName: \"kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-kube-api-access-kzpd4\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.175023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.175004 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.194065 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.194042 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:26.242809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.242778 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb"] Oct 30 23:40:26.248690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.248667 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.263864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.262818 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"distinct-2-w0-d3dcca-kserve-self-signed-certs\"" Oct 30 23:40:26.264719 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.264697 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb"] Oct 30 23:40:26.359136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.359104 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-home\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.359297 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.359184 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42fdc936-a982-454f-a538-0a6494d1cfa2-tls-certs\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.359297 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.359228 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-tmp-dir\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.359398 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.359297 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-model-cache\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.359398 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.359382 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-dshm\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.359523 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.359413 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mp2j\" (UniqueName: \"kubernetes.io/projected/42fdc936-a982-454f-a538-0a6494d1cfa2-kube-api-access-6mp2j\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.461237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.461199 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-home\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.461441 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.461303 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42fdc936-a982-454f-a538-0a6494d1cfa2-tls-certs\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.461441 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.461344 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-tmp-dir\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.461441 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.461376 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-model-cache\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.461658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.461449 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-dshm\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.461658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.461519 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mp2j\" (UniqueName: \"kubernetes.io/projected/42fdc936-a982-454f-a538-0a6494d1cfa2-kube-api-access-6mp2j\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.461856 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.461837 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-home\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.461924 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.461881 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-tmp-dir\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.461924 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.461911 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-model-cache\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.464327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.464304 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42fdc936-a982-454f-a538-0a6494d1cfa2-tls-certs\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.464618 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.464603 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-dshm\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.474170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.474137 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mp2j\" (UniqueName: \"kubernetes.io/projected/42fdc936-a982-454f-a538-0a6494d1cfa2-kube-api-access-6mp2j\") pod \"distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.496503 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:40:26.496427 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c029726_9be9_4ac8_a268_8761a4499b87.slice/crio-a2ee79c530fb41030342628f17c9a61da21420ecaeb57108dbfd16dbf5e90210 WatchSource:0}: Error finding container a2ee79c530fb41030342628f17c9a61da21420ecaeb57108dbfd16dbf5e90210: Status 404 returned error can't find the container with id a2ee79c530fb41030342628f17c9a61da21420ecaeb57108dbfd16dbf5e90210 Oct 30 23:40:26.497368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.497321 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6"] Oct 30 23:40:26.499577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.499546 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:26.499661 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.499613 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:26.499661 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.499641 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:26.579174 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.579142 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:26.669523 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.669486 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" event={"ID":"0c029726-9be9-4ac8-a268-8761a4499b87","Type":"ContainerStarted","Data":"73e9a637495396add78640f3ed8d7c33b22555c9bb55f7223f94a09d6628bc94"} Oct 30 23:40:26.669727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.669560 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" event={"ID":"0c029726-9be9-4ac8-a268-8761a4499b87","Type":"ContainerStarted","Data":"a2ee79c530fb41030342628f17c9a61da21420ecaeb57108dbfd16dbf5e90210"} Oct 30 23:40:26.703485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.699394 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" podStartSLOduration=1.699376379 podStartE2EDuration="1.699376379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:26.69376175 +0000 UTC m=+1238.212762841" watchObservedRunningTime="2026-10-30 23:40:26.699376379 +0000 UTC m=+1238.218377470" Oct 30 23:40:26.871435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:26.871412 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb"] Oct 30 23:40:26.873273 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:40:26.873247 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42fdc936_a982_454f_a538_0a6494d1cfa2.slice/crio-80f1d46443329c124eb024fe286a97f36d54b646c06b01a78aa955a1e810d63c WatchSource:0}: Error finding container 80f1d46443329c124eb024fe286a97f36d54b646c06b01a78aa955a1e810d63c: Status 404 returned error can't find the container with id 80f1d46443329c124eb024fe286a97f36d54b646c06b01a78aa955a1e810d63c Oct 30 23:40:27.206549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:27.199924 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:27.206549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:27.200063 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 23:40:27.206549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:27.200119 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" podUID="0c029726-9be9-4ac8-a268-8761a4499b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 23:40:27.678031 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:27.677988 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" event={"ID":"42fdc936-a982-454f-a538-0a6494d1cfa2","Type":"ContainerStarted","Data":"55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c"} Oct 30 23:40:27.678230 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:27.678043 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" event={"ID":"42fdc936-a982-454f-a538-0a6494d1cfa2","Type":"ContainerStarted","Data":"80f1d46443329c124eb024fe286a97f36d54b646c06b01a78aa955a1e810d63c"} Oct 30 23:40:27.711542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:27.700598 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podStartSLOduration=1.700577208 podStartE2EDuration="1.700577208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:27.697837894 +0000 UTC m=+1239.216838985" watchObservedRunningTime="2026-10-30 23:40:27.700577208 +0000 UTC m=+1239.219578300" Oct 30 23:40:28.196840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:28.196805 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 23:40:28.197424 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:28.197394 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" podUID="0c029726-9be9-4ac8-a268-8761a4499b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 23:40:28.579307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:28.579269 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:29.207486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:29.203784 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 23:40:29.207486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:29.203841 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" podUID="0c029726-9be9-4ac8-a268-8761a4499b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 23:40:29.434339 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:40:29.434300 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:40:30.205886 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:30.203562 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 23:40:30.205886 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:30.203626 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" podUID="0c029726-9be9-4ac8-a268-8761a4499b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 23:40:31.210918 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:31.210870 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 23:40:31.211363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:31.210936 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" podUID="0c029726-9be9-4ac8-a268-8761a4499b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 23:40:32.012289 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.012251 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf"] Oct 30 23:40:32.018791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.018761 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:32.020061 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.020034 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf"] Oct 30 23:40:32.023258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.023103 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:40:32.024400 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.024382 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:40:32.053167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.053146 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f53b3fe8-j5rcf\" (UID: \"bbf3579b-b533-4d34-b609-565c42d51336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:32.053274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.053197 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f53b3fe8-j5rcf\" (UID: \"bbf3579b-b533-4d34-b609-565c42d51336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:32.053334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.053280 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgsjf\" (UniqueName: \"kubernetes.io/projected/bbf3579b-b533-4d34-b609-565c42d51336-kube-api-access-zgsjf\") pod \"maas-api-revoke-sub-f53b3fe8-j5rcf\" (UID: \"bbf3579b-b533-4d34-b609-565c42d51336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:32.154547 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.154510 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f53b3fe8-j5rcf\" (UID: \"bbf3579b-b533-4d34-b609-565c42d51336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:32.154717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.154572 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f53b3fe8-j5rcf\" (UID: \"bbf3579b-b533-4d34-b609-565c42d51336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:32.154717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.154653 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgsjf\" (UniqueName: \"kubernetes.io/projected/bbf3579b-b533-4d34-b609-565c42d51336-kube-api-access-zgsjf\") pod \"maas-api-revoke-sub-f53b3fe8-j5rcf\" (UID: \"bbf3579b-b533-4d34-b609-565c42d51336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:32.155300 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.155268 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f53b3fe8-j5rcf\" (UID: \"bbf3579b-b533-4d34-b609-565c42d51336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:32.157118 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.157094 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f53b3fe8-j5rcf\" (UID: \"bbf3579b-b533-4d34-b609-565c42d51336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:32.169346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.169320 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgsjf\" (UniqueName: \"kubernetes.io/projected/bbf3579b-b533-4d34-b609-565c42d51336-kube-api-access-zgsjf\") pod \"maas-api-revoke-sub-f53b3fe8-j5rcf\" (UID: \"bbf3579b-b533-4d34-b609-565c42d51336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:32.200933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.200893 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 23:40:32.201220 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.201192 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" podUID="0c029726-9be9-4ac8-a268-8761a4499b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 23:40:32.334087 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.334062 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:32.625054 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:40:32.625025 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf3579b_b533_4d34_b609_565c42d51336.slice/crio-91e0dca0cf64c61e897a746b98d24445f38024218e633db74cc13770517a254c WatchSource:0}: Error finding container 91e0dca0cf64c61e897a746b98d24445f38024218e633db74cc13770517a254c: Status 404 returned error can't find the container with id 91e0dca0cf64c61e897a746b98d24445f38024218e633db74cc13770517a254c Oct 30 23:40:32.626156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.626127 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf"] Oct 30 23:40:32.734078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:32.734049 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" event={"ID":"bbf3579b-b533-4d34-b609-565c42d51336","Type":"ContainerStarted","Data":"91e0dca0cf64c61e897a746b98d24445f38024218e633db74cc13770517a254c"} Oct 30 23:40:33.517861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:33.517827 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:33.518586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:33.518563 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:33.520286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:33.520268 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:40:33.742572 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:33.742456 7570 generic.go:356] "Generic (PLEG): container finished" podID="bbf3579b-b533-4d34-b609-565c42d51336" containerID="3f1e5e7b3ed99157e38247533443a20f4136463a9ca50dcb3d8a75b4d7e16f3f" exitCode=0 Oct 30 23:40:33.742572 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:33.742504 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" event={"ID":"bbf3579b-b533-4d34-b609-565c42d51336","Type":"ContainerDied","Data":"3f1e5e7b3ed99157e38247533443a20f4136463a9ca50dcb3d8a75b4d7e16f3f"} Oct 30 23:40:34.900032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:34.900007 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:34.987016 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:34.986986 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-xks-ca\") pod \"bbf3579b-b533-4d34-b609-565c42d51336\" (UID: \"bbf3579b-b533-4d34-b609-565c42d51336\") " Oct 30 23:40:34.987173 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:34.987034 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-openshift-ca\") pod \"bbf3579b-b533-4d34-b609-565c42d51336\" (UID: \"bbf3579b-b533-4d34-b609-565c42d51336\") " Oct 30 23:40:34.987173 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:34.987099 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf3579b-b533-4d34-b609-565c42d51336-kube-api-access-zgsjf\" (UniqueName: \"kubernetes.io/projected/bbf3579b-b533-4d34-b609-565c42d51336-kube-api-access-zgsjf\") pod \"bbf3579b-b533-4d34-b609-565c42d51336\" (UID: \"bbf3579b-b533-4d34-b609-565c42d51336\") " Oct 30 23:40:34.987510 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:34.987484 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-openshift-ca" pod "bbf3579b-b533-4d34-b609-565c42d51336" (UID: "bbf3579b-b533-4d34-b609-565c42d51336"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:40:34.987696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:34.987677 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:34.989631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:34.989605 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-xks-ca" pod "bbf3579b-b533-4d34-b609-565c42d51336" (UID: "bbf3579b-b533-4d34-b609-565c42d51336"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:40:34.989747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:34.989702 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf3579b-b533-4d34-b609-565c42d51336-kube-api-access-zgsjf" pod "bbf3579b-b533-4d34-b609-565c42d51336" (UID: "bbf3579b-b533-4d34-b609-565c42d51336"). InnerVolumeSpecName "kube-api-access-zgsjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:35.067165 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:35.067111 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:35.084010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:35.083984 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:35.088364 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:35.088337 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf3579b-b533-4d34-b609-565c42d51336-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:35.088364 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:35.088365 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgsjf\" (UniqueName: \"kubernetes.io/projected/bbf3579b-b533-4d34-b609-565c42d51336-kube-api-access-zgsjf\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:40:35.759941 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:35.759909 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" Oct 30 23:40:35.759941 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:35.759922 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf" event={"ID":"bbf3579b-b533-4d34-b609-565c42d51336","Type":"ContainerDied","Data":"91e0dca0cf64c61e897a746b98d24445f38024218e633db74cc13770517a254c"} Oct 30 23:40:35.760199 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:35.760013 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91e0dca0cf64c61e897a746b98d24445f38024218e633db74cc13770517a254c" Oct 30 23:40:35.772858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:35.772836 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:40:36.585194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:36.584210 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:36.610052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:36.610023 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:36.835670 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:36.835267 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:40:36.837568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:36.837447 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl"] Oct 30 23:40:36.838731 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:36.838710 7570 cpu_manager.go:414] "containerMap: removing container" podUID="bbf3579b-b533-4d34-b609-565c42d51336" containerName="revoke-keys" Oct 30 23:40:36.838866 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:36.838852 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf3579b-b533-4d34-b609-565c42d51336" containerName="revoke-keys" Oct 30 23:40:36.839322 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:36.839177 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf3579b-b533-4d34-b609-565c42d51336" containerName="revoke-keys" Oct 30 23:40:36.969261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:36.969231 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl"] Oct 30 23:40:36.971918 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:36.971828 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:36.997072 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:36.997045 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"default-dockercfg-l9wjz\"" Oct 30 23:40:36.997337 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:36.997317 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"openshift-service-ca.crt\"" Oct 30 23:40:36.998214 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:36.998192 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"kube-root-ca.crt\"" Oct 30 23:40:37.030327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.029997 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"distinct-2-w6-e15304-kserve-self-signed-certs\"" Oct 30 23:40:37.052965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.038616 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6148fe-e516-492b-b5a7-755a75e82c58-tls-certs\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.052965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.038682 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmfqx\" (UniqueName: \"kubernetes.io/projected/ee6148fe-e516-492b-b5a7-755a75e82c58-kube-api-access-qmfqx\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.052965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.038754 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-dshm\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.052965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.038917 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-model-cache\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.052965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.038962 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-tmp-dir\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.052965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.039051 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-home\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.159804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.158158 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-home\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.159804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.158261 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6148fe-e516-492b-b5a7-755a75e82c58-tls-certs\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.159804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.158329 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmfqx\" (UniqueName: \"kubernetes.io/projected/ee6148fe-e516-492b-b5a7-755a75e82c58-kube-api-access-qmfqx\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.159804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.158410 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-dshm\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.159804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.159260 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-home\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.189484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.188411 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-model-cache\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.189484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.188501 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-tmp-dir\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.189484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.189282 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-model-cache\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.232384 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.232356 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-tmp-dir\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.258656 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.257829 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-dshm\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.258656 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.258420 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6148fe-e516-492b-b5a7-755a75e82c58-tls-certs\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.278505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.278157 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmfqx\" (UniqueName: \"kubernetes.io/projected/ee6148fe-e516-492b-b5a7-755a75e82c58-kube-api-access-qmfqx\") pod \"distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:37.564924 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:37.564829 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:38.489454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:38.489369 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl"] Oct 30 23:40:38.849412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:38.849374 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" event={"ID":"ee6148fe-e516-492b-b5a7-755a75e82c58","Type":"ContainerStarted","Data":"4d5e41c0d0ab89419a20ad13530d6b08ee78346448059542a0475a7805ffd355"} Oct 30 23:40:38.849603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:38.849434 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" event={"ID":"ee6148fe-e516-492b-b5a7-755a75e82c58","Type":"ContainerStarted","Data":"fd5dcc61353c067fac281a25326b737d1e5496fa9cc24805ecf7a72cadb88df2"} Oct 30 23:40:38.872808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:38.872759 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podStartSLOduration=2.872745787 podStartE2EDuration="2.872745787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:38.870346021 +0000 UTC m=+1250.389347224" watchObservedRunningTime="2026-10-30 23:40:38.872745787 +0000 UTC m=+1250.391746876" Oct 30 23:40:39.571276 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:39.565543 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:40.999095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:40.999058 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24"] Oct 30 23:40:41.004493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.004434 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.023351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.023320 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24"] Oct 30 23:40:41.074151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.074112 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.074312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.074175 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdnh9\" (UniqueName: \"kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-kube-api-access-mdnh9\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.074312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.074268 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.074389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.074313 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.074437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.074419 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.074511 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.074479 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.074590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.074578 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.074628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.074602 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe9d9e-818f-4909-a48e-555b42174d32-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.074659 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.074645 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.176423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.176385 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdnh9\" (UniqueName: \"kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-kube-api-access-mdnh9\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.176603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.176434 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.176603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.176478 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.176603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.176530 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.176603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.176561 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.176851 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.176620 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.176851 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.176647 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe9d9e-818f-4909-a48e-555b42174d32-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.176851 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.176720 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.176851 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.176845 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.177047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.176954 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.177047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.176997 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.177396 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.177368 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.177598 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.177408 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.177739 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.177718 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe9d9e-818f-4909-a48e-555b42174d32-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.179210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.179191 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.179534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.179511 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.188931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.188907 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.189398 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.189377 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdnh9\" (UniqueName: \"kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-kube-api-access-mdnh9\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.326882 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.326804 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:41.616990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.616962 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24"] Oct 30 23:40:41.617805 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:40:41.617780 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfe9d9e_818f_4909_a48e_555b42174d32.slice/crio-19c929b25349428aa790ca140d26837c1d0d53c3206915f68c25d986ac7568e2 WatchSource:0}: Error finding container 19c929b25349428aa790ca140d26837c1d0d53c3206915f68c25d986ac7568e2: Status 404 returned error can't find the container with id 19c929b25349428aa790ca140d26837c1d0d53c3206915f68c25d986ac7568e2 Oct 30 23:40:41.620354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.620319 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:41.620427 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.620388 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:41.620427 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.620416 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:41.885571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.885437 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" event={"ID":"5dfe9d9e-818f-4909-a48e-555b42174d32","Type":"ContainerStarted","Data":"162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1"} Oct 30 23:40:41.885571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.885546 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" event={"ID":"5dfe9d9e-818f-4909-a48e-555b42174d32","Type":"ContainerStarted","Data":"19c929b25349428aa790ca140d26837c1d0d53c3206915f68c25d986ac7568e2"} Oct 30 23:40:41.911083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:41.911040 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" podStartSLOduration=1.911026514 podStartE2EDuration="1.911026514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:41.910239691 +0000 UTC m=+1253.429240793" watchObservedRunningTime="2026-10-30 23:40:41.911026514 +0000 UTC m=+1253.430027622" Oct 30 23:40:42.327769 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:42.327735 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:42.329540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:42.329516 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:40:42.329667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:42.329561 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 23:40:43.327787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:43.327746 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:40:43.329233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:43.328598 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 23:40:44.327598 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:44.327567 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:40:44.327810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:44.327618 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 23:40:45.328007 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:45.327968 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:40:45.328595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:45.328028 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 23:40:46.327340 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:46.327306 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:40:46.327552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:46.327362 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 23:40:47.327961 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:47.327927 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:40:47.328569 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:47.327993 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 23:40:47.565995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:47.565959 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:47.579130 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:47.579048 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:47.963168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:47.963080 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:40:48.327555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:48.327446 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:40:48.327555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:48.327521 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 23:40:48.840420 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:48.840384 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r"] Oct 30 23:40:48.868619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:48.868586 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r"] Oct 30 23:40:48.868867 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:48.868773 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:48.875783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:48.875752 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"openshift-service-ca.crt\"" Oct 30 23:40:48.877111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:48.877090 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"kube-root-ca.crt\"" Oct 30 23:40:48.877864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:48.877735 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"distinct-2-w5-dd3a15-kserve-self-signed-certs\"" Oct 30 23:40:48.877864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:48.877836 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"default-dockercfg-ftrjj\"" Oct 30 23:40:49.000130 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.000095 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-home\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.000321 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.000174 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-dshm\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.000321 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.000215 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwxh\" (UniqueName: \"kubernetes.io/projected/809bef14-259b-4a21-b170-a6a701e00e6c-kube-api-access-dfwxh\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.000493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.000450 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809bef14-259b-4a21-b170-a6a701e00e6c-tls-certs\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.000647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.000557 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-model-cache\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.000647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.000582 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-tmp-dir\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.101959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.101869 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-home\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.101959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.101933 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-dshm\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.101959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.101951 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwxh\" (UniqueName: \"kubernetes.io/projected/809bef14-259b-4a21-b170-a6a701e00e6c-kube-api-access-dfwxh\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.102254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.102021 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809bef14-259b-4a21-b170-a6a701e00e6c-tls-certs\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.102254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.102073 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-model-cache\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.102254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.102105 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-tmp-dir\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.102397 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.102313 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-home\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.102397 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.102382 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-model-cache\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.102533 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.102514 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-tmp-dir\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.104572 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.104553 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-dshm\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.104864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.104845 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809bef14-259b-4a21-b170-a6a701e00e6c-tls-certs\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.114894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.114860 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwxh\" (UniqueName: \"kubernetes.io/projected/809bef14-259b-4a21-b170-a6a701e00e6c-kube-api-access-dfwxh\") pod \"distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.190885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.190851 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:49.331982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.331951 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:49.332604 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.332578 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:49.333406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.333385 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:40:49.477767 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:40:49.477724 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod809bef14_259b_4a21_b170_a6a701e00e6c.slice/crio-178cf9eaeaccbae96aa052845e91cddb36b2fe3882ff181d6bfff122449379d3 WatchSource:0}: Error finding container 178cf9eaeaccbae96aa052845e91cddb36b2fe3882ff181d6bfff122449379d3: Status 404 returned error can't find the container with id 178cf9eaeaccbae96aa052845e91cddb36b2fe3882ff181d6bfff122449379d3 Oct 30 23:40:49.477889 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.477849 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r"] Oct 30 23:40:49.967963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.967923 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" event={"ID":"809bef14-259b-4a21-b170-a6a701e00e6c","Type":"ContainerStarted","Data":"1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca"} Oct 30 23:40:49.967963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.967979 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" event={"ID":"809bef14-259b-4a21-b170-a6a701e00e6c","Type":"ContainerStarted","Data":"178cf9eaeaccbae96aa052845e91cddb36b2fe3882ff181d6bfff122449379d3"} Oct 30 23:40:49.989273 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:49.989224 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podStartSLOduration=1.989211116 podStartE2EDuration="1.989211116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:49.987494646 +0000 UTC m=+1261.506495736" watchObservedRunningTime="2026-10-30 23:40:49.989211116 +0000 UTC m=+1261.508212205" Oct 30 23:40:50.191700 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:50.191669 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:55.663873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.663838 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj"] Oct 30 23:40:55.668589 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.668560 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.684088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.684054 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj"] Oct 30 23:40:55.794832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.782728 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n"] Oct 30 23:40:55.794832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.787756 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:55.812112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.812079 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.812325 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.812135 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.812325 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.812183 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.812325 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.812230 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.812325 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.812264 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.812325 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.812348 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.812659 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.812624 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.812741 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.812718 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.812882 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.812763 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rtwn\" (UniqueName: \"kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-kube-api-access-4rtwn\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.816510 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.816452 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n"] Oct 30 23:40:55.914050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.913959 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.914050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914009 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:55.914237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914118 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:55.914237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914180 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:55.914302 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914264 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:55.914302 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914287 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.914365 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914357 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rtwn\" (UniqueName: \"kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-kube-api-access-4rtwn\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.914398 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914380 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.914432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914398 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.914500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914426 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.914500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914444 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:55.914595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914555 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.914745 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914723 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:55.914806 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914764 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.914806 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914774 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.914887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914808 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:55.914887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914844 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4vx7\" (UniqueName: \"kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-kube-api-access-q4vx7\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:55.914887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914867 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.914982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914942 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:55.914982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.914968 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.915047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.915003 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.915047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.915029 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.915331 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.915309 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.917151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.917130 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.917507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.917451 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.926788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.926764 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rtwn\" (UniqueName: \"kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-kube-api-access-4rtwn\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.926876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.926765 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:55.990710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:55.990683 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:56.016089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.016053 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.016236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.016118 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.016236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.016142 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.016338 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.016246 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.016388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.016336 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.016550 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.016530 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.016635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.016572 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.016635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.016596 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4vx7\" (UniqueName: \"kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-kube-api-access-q4vx7\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.016741 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.016634 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.016741 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.016637 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.016741 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.016708 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.016949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.016923 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.017108 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.017083 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.017171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.017109 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.019282 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.019253 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.021031 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.021009 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.031401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.031379 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.032093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.032077 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4vx7\" (UniqueName: \"kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-kube-api-access-q4vx7\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.084168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.084133 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l"] Oct 30 23:40:56.090430 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.090406 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.102507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.101767 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l"] Oct 30 23:40:56.119954 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.119926 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:56.220179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.220140 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.220826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.220238 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.220826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.220295 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf77418c-72fd-40ad-b389-df9273f36cfc-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.220826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.220426 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.220826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.220491 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.220826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.220542 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qswdq\" (UniqueName: \"kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-kube-api-access-qswdq\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.220826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.220596 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.220826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.220626 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.220826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.220664 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.282963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.282927 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj"] Oct 30 23:40:56.283092 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:40:56.283001 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cdbcc63_7336_46ed_99bc_0e3386f53da5.slice/crio-804a09e7e978f598f9971e63491ec11b5ab90aaee30810692b5479c00794c197 WatchSource:0}: Error finding container 804a09e7e978f598f9971e63491ec11b5ab90aaee30810692b5479c00794c197: Status 404 returned error can't find the container with id 804a09e7e978f598f9971e63491ec11b5ab90aaee30810692b5479c00794c197 Oct 30 23:40:56.285676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.285640 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:56.285786 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.285722 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:56.285786 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.285767 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:56.321719 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.321690 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.321846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.321814 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.321908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.321865 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.321981 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.321907 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf77418c-72fd-40ad-b389-df9273f36cfc-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.322033 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.322012 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.322092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.322052 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.322156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.322138 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.322212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.322180 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.322285 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.322260 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qswdq\" (UniqueName: \"kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-kube-api-access-qswdq\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.322366 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.322351 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.322408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.322386 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.322449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.322352 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.322449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.322418 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.322723 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.322706 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf77418c-72fd-40ad-b389-df9273f36cfc-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.324624 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.324605 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.324838 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.324821 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.335390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.335368 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qswdq\" (UniqueName: \"kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-kube-api-access-qswdq\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.335562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.335541 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.415440 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.415414 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n"] Oct 30 23:40:56.416248 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:40:56.416224 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42ff644_b417_4c6e_9ecf_aea24c6233d9.slice/crio-7e3d2667ce714f6c56c54e0221686694e966ba8f20287ceeab5c26ca55ad96c0 WatchSource:0}: Error finding container 7e3d2667ce714f6c56c54e0221686694e966ba8f20287ceeab5c26ca55ad96c0: Status 404 returned error can't find the container with id 7e3d2667ce714f6c56c54e0221686694e966ba8f20287ceeab5c26ca55ad96c0 Oct 30 23:40:56.419700 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.419655 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:56.419818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.419743 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:56.419818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.419786 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:56.420067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.420044 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:56.733501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.733430 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l"] Oct 30 23:40:56.735011 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:40:56.734971 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf77418c_72fd_40ad_b389_df9273f36cfc.slice/crio-56cfcfa4e3a05425fd1c4bc22f832e661418b9535ed0d26de65b6ad8a4377f70 WatchSource:0}: Error finding container 56cfcfa4e3a05425fd1c4bc22f832e661418b9535ed0d26de65b6ad8a4377f70: Status 404 returned error can't find the container with id 56cfcfa4e3a05425fd1c4bc22f832e661418b9535ed0d26de65b6ad8a4377f70 Oct 30 23:40:56.738090 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.738053 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:56.738173 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.738153 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:56.738224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:56.738193 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:40:57.024938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.024834 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" event={"ID":"d42ff644-b417-4c6e-9ecf-aea24c6233d9","Type":"ContainerStarted","Data":"37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb"} Oct 30 23:40:57.024938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.024905 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" event={"ID":"d42ff644-b417-4c6e-9ecf-aea24c6233d9","Type":"ContainerStarted","Data":"7e3d2667ce714f6c56c54e0221686694e966ba8f20287ceeab5c26ca55ad96c0"} Oct 30 23:40:57.026895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.026865 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" event={"ID":"0cdbcc63-7336-46ed-99bc-0e3386f53da5","Type":"ContainerStarted","Data":"5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f"} Oct 30 23:40:57.027023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.026915 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" event={"ID":"0cdbcc63-7336-46ed-99bc-0e3386f53da5","Type":"ContainerStarted","Data":"804a09e7e978f598f9971e63491ec11b5ab90aaee30810692b5479c00794c197"} Oct 30 23:40:57.028627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.028589 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" event={"ID":"cf77418c-72fd-40ad-b389-df9273f36cfc","Type":"ContainerStarted","Data":"50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e"} Oct 30 23:40:57.028757 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.028650 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" event={"ID":"cf77418c-72fd-40ad-b389-df9273f36cfc","Type":"ContainerStarted","Data":"56cfcfa4e3a05425fd1c4bc22f832e661418b9535ed0d26de65b6ad8a4377f70"} Oct 30 23:40:57.050435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.050376 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podStartSLOduration=2.050359373 podStartE2EDuration="2.050359373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:57.048406969 +0000 UTC m=+1268.567408059" watchObservedRunningTime="2026-10-30 23:40:57.050359373 +0000 UTC m=+1268.569360463" Oct 30 23:40:57.073629 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.073568 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podStartSLOduration=1.073551707 podStartE2EDuration="1.073551707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:57.072786073 +0000 UTC m=+1268.591787166" watchObservedRunningTime="2026-10-30 23:40:57.073551707 +0000 UTC m=+1268.592552797" Oct 30 23:40:57.101203 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.101147 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" podStartSLOduration=2.10113297 podStartE2EDuration="2.10113297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:57.098705655 +0000 UTC m=+1268.617706746" watchObservedRunningTime="2026-10-30 23:40:57.10113297 +0000 UTC m=+1268.620134059" Oct 30 23:40:57.121114 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.121083 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:40:57.122624 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.122594 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:40:57.122778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.122656 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 23:40:57.424539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.420903 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:40:57.433490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.427598 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:40:57.433490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.427662 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 23:40:57.995488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:57.993818 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:40:58.005504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:58.003617 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:40:58.005504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:58.003682 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 23:40:58.143489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:58.124273 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:40:58.143489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:58.124334 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 23:40:58.432489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:58.425692 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:40:58.432489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:58.425760 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 23:40:59.031407 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:59.031363 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:40:59.031945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:59.031434 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 23:40:59.142481 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:59.126921 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:40:59.142481 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:59.126999 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 23:40:59.194489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:59.193903 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:59.283095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:59.282946 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:40:59.445532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:59.444862 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:40:59.445532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:59.444929 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 23:40:59.569680 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:40:59.568934 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:40:59.998524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:59.991295 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:40:59.998524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:40:59.991358 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 23:41:00.124489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:00.123958 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:41:00.124489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:00.124021 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 23:41:00.382129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:00.380426 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:41:00.426489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:00.421117 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:41:00.426489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:00.421173 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 23:41:01.003485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:01.003336 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:41:01.003485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:01.003400 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 23:41:01.136613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:01.136577 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:41:01.137205 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:01.137175 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 23:41:01.436485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:01.428135 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:41:01.436485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:01.428206 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 23:41:01.571489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:01.568565 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:01.629489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:01.628558 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:01.781542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:01.779565 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:01.792485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:01.791552 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:01.956489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:01.955399 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:01.996490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:01.978558 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:02.010647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.010609 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:41:02.012650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.012613 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 23:41:02.123880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.123841 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:41:02.124168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.124137 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 23:41:02.217615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.217562 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:02.402159 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.402057 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:02.420966 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.420929 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:41:02.421287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.421238 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 23:41:02.553486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.533563 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:02.597487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.570556 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:02.597487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.571393 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:41:02.735968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.698935 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:02.752507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.747192 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:41:02.910554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.782578 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:02.910554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.785025 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:41:02.910554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.785089 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:41:02.910554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.882547 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:02.910554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.884127 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:41:03.002595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.964932 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:03.002595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:02.975324 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:41:03.010497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.010320 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:03.079750 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.015296 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:41:03.079750 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.044046 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:41:03.079750 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.044109 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 23:41:03.156210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.155597 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:41:03.156210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.155661 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 23:41:03.239753 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.239699 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:03.264603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.245567 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:41:03.458024 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.457879 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:41:03.659529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.463765 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:41:03.659529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.641819 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:41:03.803815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.803750 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:41:03.839911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.839862 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:41:03.937723 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.862372 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:41:04.025809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:03.950096 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:04.026290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.026261 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:41:04.026581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.026550 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:04.114490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.025588 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:41:04.497671 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.455400 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:41:04.803615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.684097 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:04.803615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.508088 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:41:04.803615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.533112 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:04.803615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.780592 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:04.803615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.508052 7570 trace.go:236] Trace[2002979983]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" (30-Oct-2026 23:41:03.173) (total time: 1334ms): Oct 30 23:41:04.803615 ip-10-0-137-81 kubenswrapper[7570]: Trace[2002979983]: [1.334839974s] [1.334839974s] END Oct 30 23:41:04.803615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.533055 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:41:04.829520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.586653 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:41:05.049489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.807622 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.059541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.832213 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.069487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.834736 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.069487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.894578 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:41:05.069487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.918612 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.078660 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.079053 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.918752 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.111725 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.989773 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.112515 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.782550 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.112639 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.016841 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.112981 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:04.609411 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.113041 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" podUID="0c029726-9be9-4ac8-a268-8761a4499b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.138504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.113098 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.139883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.139842 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.145731 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.145688 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.166489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.146721 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:41:05.286621 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.286572 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 23:41:05.302490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.222146 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.339562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.148012 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:41:05.339562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.222066 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.339562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.209387 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:41:05.339562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.307968 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 23:41:05.339562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.307927 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 23:41:05.490825 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.490775 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.554211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.554164 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.610358 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.604603 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:41:05.610674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.610623 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 23:41:05.694646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.630566 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:41:05.698618 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.698548 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.787489 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:41:05.786135 7570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.188s" Oct 30 23:41:05.787489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.786240 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:41:05.787489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.786379 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:41:05.825489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.816537 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.904546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.897091 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:05.982505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:05.981870 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:41:06.077333 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.077287 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:06.143569 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.082537 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:41:06.143569 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.082575 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:41:06.143569 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.082595 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:41:06.143569 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.139535 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:41:06.146794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.145709 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:41:06.173081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.147067 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:06.173081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.158762 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:41:06.173081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.158815 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 23:41:06.173081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.158930 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:41:06.173081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.158959 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 23:41:06.193521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.190521 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:41:06.193521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.190607 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:06.438533 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.360173 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:41:06.438533 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.381100 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:41:06.438533 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.383895 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:41:06.615481 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.614651 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:41:06.615481 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.614708 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 23:41:06.642958 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.642920 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:06.730523 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.730420 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:41:06.730767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.730751 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:41:06.730924 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.730909 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:41:06.731058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.731045 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:41:06.731187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.731176 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:41:06.731332 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.731319 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:41:06.731479 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.731452 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:41:06.732380 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.732362 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:41:06.732574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:06.732560 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:41:07.001233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.000895 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:41:07.001547 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.001517 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 23:41:07.142023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.141983 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:41:07.142313 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.142277 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 23:41:07.190490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.179209 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9"] Oct 30 23:41:07.242388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.242354 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9"] Oct 30 23:41:07.245549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.243585 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:07.252253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.252190 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:41:07.252253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.252194 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:41:07.311736 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.311695 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c5cb38a-7v9w9\" (UID: \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:07.312099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.312050 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c5cb38a-7v9w9\" (UID: \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:07.312534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.312508 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsscf\" (UniqueName: \"kubernetes.io/projected/aa11a2b2-7ab3-405c-bd19-01499d1867ed-kube-api-access-qsscf\") pod \"maas-api-revoke-sub-2c5cb38a-7v9w9\" (UID: \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:07.415486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.414665 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c5cb38a-7v9w9\" (UID: \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:07.419076 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.415863 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c5cb38a-7v9w9\" (UID: \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:07.419076 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.415947 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c5cb38a-7v9w9\" (UID: \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:07.419076 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.416121 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsscf\" (UniqueName: \"kubernetes.io/projected/aa11a2b2-7ab3-405c-bd19-01499d1867ed-kube-api-access-qsscf\") pod \"maas-api-revoke-sub-2c5cb38a-7v9w9\" (UID: \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:07.421772 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.421747 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c5cb38a-7v9w9\" (UID: \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:07.421939 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.421789 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:41:07.422105 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.422078 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 23:41:07.450296 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.450268 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsscf\" (UniqueName: \"kubernetes.io/projected/aa11a2b2-7ab3-405c-bd19-01499d1867ed-kube-api-access-qsscf\") pod \"maas-api-revoke-sub-2c5cb38a-7v9w9\" (UID: \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:07.566523 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.566403 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:07.984443 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:07.983053 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9"] Oct 30 23:41:08.007037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:08.003881 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:41:08.007037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:08.005607 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:41:08.007037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:08.006996 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:41:08.946566 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:08.946530 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" event={"ID":"aa11a2b2-7ab3-405c-bd19-01499d1867ed","Type":"ContainerStarted","Data":"0bf02f0a1708240c4b1d5393e05de936d4cc65b7ebf4f642cb056bdab31fe987"} Oct 30 23:41:08.946963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:08.946933 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" event={"ID":"aa11a2b2-7ab3-405c-bd19-01499d1867ed","Type":"ContainerStarted","Data":"5f2464b5e3cbee90fd47d9c50815b1ab5a2020662592a16179fdd2d22a831b06"} Oct 30 23:41:09.000530 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:08.997694 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" podStartSLOduration=1.9976705639999999 podStartE2EDuration="1.997670564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:08.987734069 +0000 UTC m=+1280.506735161" watchObservedRunningTime="2026-10-30 23:41:08.997670564 +0000 UTC m=+1280.516671652" Oct 30 23:41:09.174644 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:09.174597 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:41:09.174859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:09.174657 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:09.350912 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:09.350880 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:41:09.352949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:09.352927 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:41:09.362386 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:09.360788 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:41:09.475658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:09.475618 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:41:09.479971 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:09.479919 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:09.867402 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:09.867374 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:41:09.867776 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:09.867755 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:41:09.868657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:09.868633 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:41:09.956917 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:09.956874 7570 generic.go:356] "Generic (PLEG): container finished" podID="aa11a2b2-7ab3-405c-bd19-01499d1867ed" containerID="0bf02f0a1708240c4b1d5393e05de936d4cc65b7ebf4f642cb056bdab31fe987" exitCode=0 Oct 30 23:41:09.957280 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:09.957247 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" event={"ID":"aa11a2b2-7ab3-405c-bd19-01499d1867ed","Type":"ContainerDied","Data":"0bf02f0a1708240c4b1d5393e05de936d4cc65b7ebf4f642cb056bdab31fe987"} Oct 30 23:41:11.144758 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.144732 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:11.201792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.201752 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-xks-ca\") pod \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\" (UID: \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\") " Oct 30 23:41:11.201982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.201817 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa11a2b2-7ab3-405c-bd19-01499d1867ed-kube-api-access-qsscf\" (UniqueName: \"kubernetes.io/projected/aa11a2b2-7ab3-405c-bd19-01499d1867ed-kube-api-access-qsscf\") pod \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\" (UID: \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\") " Oct 30 23:41:11.201982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.201927 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-openshift-ca\") pod \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\" (UID: \"aa11a2b2-7ab3-405c-bd19-01499d1867ed\") " Oct 30 23:41:11.202573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.202362 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-openshift-ca" pod "aa11a2b2-7ab3-405c-bd19-01499d1867ed" (UID: "aa11a2b2-7ab3-405c-bd19-01499d1867ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:41:11.202703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.202654 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:11.204381 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.204348 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-xks-ca" pod "aa11a2b2-7ab3-405c-bd19-01499d1867ed" (UID: "aa11a2b2-7ab3-405c-bd19-01499d1867ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:41:11.204578 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.204557 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa11a2b2-7ab3-405c-bd19-01499d1867ed-kube-api-access-qsscf" pod "aa11a2b2-7ab3-405c-bd19-01499d1867ed" (UID: "aa11a2b2-7ab3-405c-bd19-01499d1867ed"). InnerVolumeSpecName "kube-api-access-qsscf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:41:11.304031 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.303942 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa11a2b2-7ab3-405c-bd19-01499d1867ed-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:11.304031 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.303975 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsscf\" (UniqueName: \"kubernetes.io/projected/aa11a2b2-7ab3-405c-bd19-01499d1867ed-kube-api-access-qsscf\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:11.951058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.951028 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn"] Oct 30 23:41:11.951788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.951760 7570 cpu_manager.go:414] "containerMap: removing container" podUID="aa11a2b2-7ab3-405c-bd19-01499d1867ed" containerName="revoke-keys" Oct 30 23:41:11.951788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.951778 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa11a2b2-7ab3-405c-bd19-01499d1867ed" containerName="revoke-keys" Oct 30 23:41:11.951956 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.951890 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa11a2b2-7ab3-405c-bd19-01499d1867ed" containerName="revoke-keys" Oct 30 23:41:11.955376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.955360 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:11.970188 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.968335 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"distinct-2-w4-843d8c-kserve-self-signed-certs\"" Oct 30 23:41:11.974667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.974631 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn"] Oct 30 23:41:11.981030 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.981006 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" Oct 30 23:41:11.981153 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.981036 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9" event={"ID":"aa11a2b2-7ab3-405c-bd19-01499d1867ed","Type":"ContainerDied","Data":"5f2464b5e3cbee90fd47d9c50815b1ab5a2020662592a16179fdd2d22a831b06"} Oct 30 23:41:11.981153 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:11.981083 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2464b5e3cbee90fd47d9c50815b1ab5a2020662592a16179fdd2d22a831b06" Oct 30 23:41:12.014879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.014847 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpngw\" (UniqueName: \"kubernetes.io/projected/d073cc7c-6856-4131-8bf7-250400f05d7f-kube-api-access-tpngw\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.015040 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.015025 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-dshm\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.015110 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.015080 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-model-cache\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.015242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.015225 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-tmp-dir\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.015324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.015249 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-home\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.015324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.015306 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d073cc7c-6856-4131-8bf7-250400f05d7f-tls-certs\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.117350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.116906 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpngw\" (UniqueName: \"kubernetes.io/projected/d073cc7c-6856-4131-8bf7-250400f05d7f-kube-api-access-tpngw\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.117350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.117056 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-dshm\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.117350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.117128 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-model-cache\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.119813 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.118638 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-tmp-dir\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.119813 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.118675 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-home\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.119813 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.118749 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d073cc7c-6856-4131-8bf7-250400f05d7f-tls-certs\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.119813 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.118943 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-model-cache\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.120893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.120869 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-tmp-dir\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.121302 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.121241 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-home\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.132139 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.132080 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d073cc7c-6856-4131-8bf7-250400f05d7f-tls-certs\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.134666 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.134621 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpngw\" (UniqueName: \"kubernetes.io/projected/d073cc7c-6856-4131-8bf7-250400f05d7f-kube-api-access-tpngw\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.135924 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.135903 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-dshm\") pod \"distinct-2-w4-843d8c-kserve-8644666f64-vhwnn\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.279884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.279793 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:12.577719 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.577690 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn"] Oct 30 23:41:12.990373 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.990342 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" event={"ID":"d073cc7c-6856-4131-8bf7-250400f05d7f","Type":"ContainerStarted","Data":"72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98"} Oct 30 23:41:12.990542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:12.990382 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" event={"ID":"d073cc7c-6856-4131-8bf7-250400f05d7f","Type":"ContainerStarted","Data":"b2e2fbb04e1dcc3a710526081b32e9910f1590b40e6e6cce6ecbdc6684ed9241"} Oct 30 23:41:13.039223 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:13.039173 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podStartSLOduration=2.039159163 podStartE2EDuration="2.039159163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:13.037733539 +0000 UTC m=+1284.556734630" watchObservedRunningTime="2026-10-30 23:41:13.039159163 +0000 UTC m=+1284.558160252" Oct 30 23:41:13.281510 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:13.281416 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:14.671899 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.671865 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294"] Oct 30 23:41:14.676199 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.676178 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:14.685003 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.684950 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:41:14.685485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.685207 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:41:14.690374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.690347 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294"] Oct 30 23:41:14.762281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.762248 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn6g2\" (UniqueName: \"kubernetes.io/projected/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-kube-api-access-vn6g2\") pod \"maas-api-revoke-sub-1a72cba9-m6294\" (UID: \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:14.762492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.762447 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a72cba9-m6294\" (UID: \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:14.762556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.762491 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a72cba9-m6294\" (UID: \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:14.865109 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.864297 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn6g2\" (UniqueName: \"kubernetes.io/projected/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-kube-api-access-vn6g2\") pod \"maas-api-revoke-sub-1a72cba9-m6294\" (UID: \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:14.865109 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.864597 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a72cba9-m6294\" (UID: \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:14.865109 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.864640 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a72cba9-m6294\" (UID: \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:14.866266 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.866148 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a72cba9-m6294\" (UID: \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:14.867858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.867831 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a72cba9-m6294\" (UID: \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:14.880409 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.880348 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn6g2\" (UniqueName: \"kubernetes.io/projected/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-kube-api-access-vn6g2\") pod \"maas-api-revoke-sub-1a72cba9-m6294\" (UID: \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:14.993846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:14.993767 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:15.284618 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:15.284588 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294"] Oct 30 23:41:15.285046 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:41:15.285024 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de3ab3b_0dca_43b2_b401_ba8f35dc5d3b.slice/crio-708ff107350c6078b7c4406474b0b898fb3ab8c62dd0f67c5fc71e1fab6a4a98 WatchSource:0}: Error finding container 708ff107350c6078b7c4406474b0b898fb3ab8c62dd0f67c5fc71e1fab6a4a98: Status 404 returned error can't find the container with id 708ff107350c6078b7c4406474b0b898fb3ab8c62dd0f67c5fc71e1fab6a4a98 Oct 30 23:41:16.013131 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:16.013035 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" event={"ID":"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b","Type":"ContainerStarted","Data":"1fc90eaf4f6f93b847311fc784f05e7da646fef533cf66e4c37ac13777a82cda"} Oct 30 23:41:16.013131 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:16.013102 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" event={"ID":"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b","Type":"ContainerStarted","Data":"708ff107350c6078b7c4406474b0b898fb3ab8c62dd0f67c5fc71e1fab6a4a98"} Oct 30 23:41:16.033116 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:16.033063 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" podStartSLOduration=2.033051491 podStartE2EDuration="2.033051491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:16.032441438 +0000 UTC m=+1287.551442529" watchObservedRunningTime="2026-10-30 23:41:16.033051491 +0000 UTC m=+1287.552052580" Oct 30 23:41:17.021957 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:17.021923 7570 generic.go:356] "Generic (PLEG): container finished" podID="9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b" containerID="1fc90eaf4f6f93b847311fc784f05e7da646fef533cf66e4c37ac13777a82cda" exitCode=0 Oct 30 23:41:17.022380 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:17.022010 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" event={"ID":"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b","Type":"ContainerDied","Data":"1fc90eaf4f6f93b847311fc784f05e7da646fef533cf66e4c37ac13777a82cda"} Oct 30 23:41:18.221580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.221535 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:18.329990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.329957 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-openshift-ca\") pod \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\" (UID: \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\") " Oct 30 23:41:18.330203 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.330188 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-xks-ca\") pod \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\" (UID: \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\") " Oct 30 23:41:18.330288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.330276 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-kube-api-access-vn6g2\" (UniqueName: \"kubernetes.io/projected/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-kube-api-access-vn6g2\") pod \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\" (UID: \"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b\") " Oct 30 23:41:18.330390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.330363 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-openshift-ca" pod "9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b" (UID: "9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:41:18.330781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.330758 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:18.332655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.332628 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-xks-ca" pod "9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b" (UID: "9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:41:18.332795 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.332771 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-kube-api-access-vn6g2" pod "9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b" (UID: "9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b"). InnerVolumeSpecName "kube-api-access-vn6g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:41:18.432190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.432150 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:18.432374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.432214 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn6g2\" (UniqueName: \"kubernetes.io/projected/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b-kube-api-access-vn6g2\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:18.677992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.677957 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp"] Oct 30 23:41:18.678724 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.678702 7570 cpu_manager.go:414] "containerMap: removing container" podUID="9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b" containerName="revoke-keys" Oct 30 23:41:18.678724 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.678725 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b" containerName="revoke-keys" Oct 30 23:41:18.678930 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.678916 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b" containerName="revoke-keys" Oct 30 23:41:18.682658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.682637 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:18.731835 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.731806 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp"] Oct 30 23:41:18.838618 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.838582 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcxp2\" (UniqueName: \"kubernetes.io/projected/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-kube-api-access-wcxp2\") pod \"maas-api-revoke-sub-6e2373f5-8l4hp\" (UID: \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:18.838807 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.838719 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e2373f5-8l4hp\" (UID: \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:18.838807 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.838759 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e2373f5-8l4hp\" (UID: \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:18.939999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.939917 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcxp2\" (UniqueName: \"kubernetes.io/projected/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-kube-api-access-wcxp2\") pod \"maas-api-revoke-sub-6e2373f5-8l4hp\" (UID: \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:18.939999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.939997 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e2373f5-8l4hp\" (UID: \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:18.940189 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.940026 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e2373f5-8l4hp\" (UID: \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:18.940713 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.940689 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e2373f5-8l4hp\" (UID: \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:18.942523 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.942504 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e2373f5-8l4hp\" (UID: \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:18.955156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.955129 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcxp2\" (UniqueName: \"kubernetes.io/projected/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-kube-api-access-wcxp2\") pod \"maas-api-revoke-sub-6e2373f5-8l4hp\" (UID: \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:18.999293 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:18.999258 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:19.050985 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:19.050957 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" Oct 30 23:41:19.051166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:19.051002 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294" event={"ID":"9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b","Type":"ContainerDied","Data":"708ff107350c6078b7c4406474b0b898fb3ab8c62dd0f67c5fc71e1fab6a4a98"} Oct 30 23:41:19.051166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:19.051057 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="708ff107350c6078b7c4406474b0b898fb3ab8c62dd0f67c5fc71e1fab6a4a98" Oct 30 23:41:19.284034 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:19.284008 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp"] Oct 30 23:41:19.284539 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:41:19.284517 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79b52a7_5ecd_40f1_a52f_bd5f82fb2f8c.slice/crio-a668e892f0dd78a1b6993151fbb5ea62e77f4037df7be65f36eb1cf30fa3977f WatchSource:0}: Error finding container a668e892f0dd78a1b6993151fbb5ea62e77f4037df7be65f36eb1cf30fa3977f: Status 404 returned error can't find the container with id a668e892f0dd78a1b6993151fbb5ea62e77f4037df7be65f36eb1cf30fa3977f Oct 30 23:41:20.060042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:20.059956 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" event={"ID":"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c","Type":"ContainerStarted","Data":"cb8d31452d461f987a167ab59413f5ab2deac9f0cb4c1b8020a9c67c2ad25258"} Oct 30 23:41:20.060042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:20.060013 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" event={"ID":"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c","Type":"ContainerStarted","Data":"a668e892f0dd78a1b6993151fbb5ea62e77f4037df7be65f36eb1cf30fa3977f"} Oct 30 23:41:20.082847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:20.082789 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" podStartSLOduration=2.082774739 podStartE2EDuration="2.082774739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:20.079423912 +0000 UTC m=+1291.598425002" watchObservedRunningTime="2026-10-30 23:41:20.082774739 +0000 UTC m=+1291.601775828" Oct 30 23:41:21.069025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:21.068990 7570 generic.go:356] "Generic (PLEG): container finished" podID="e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c" containerID="cb8d31452d461f987a167ab59413f5ab2deac9f0cb4c1b8020a9c67c2ad25258" exitCode=0 Oct 30 23:41:21.069486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:21.069083 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" event={"ID":"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c","Type":"ContainerDied","Data":"cb8d31452d461f987a167ab59413f5ab2deac9f0cb4c1b8020a9c67c2ad25258"} Oct 30 23:41:22.229516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:22.229492 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:22.281357 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:22.281312 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:22.298083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:22.298054 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:22.394512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:22.394479 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-xks-ca\") pod \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\" (UID: \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\") " Oct 30 23:41:22.394667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:22.394641 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-openshift-ca\") pod \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\" (UID: \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\") " Oct 30 23:41:22.394667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:22.394662 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-kube-api-access-wcxp2\" (UniqueName: \"kubernetes.io/projected/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-kube-api-access-wcxp2\") pod \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\" (UID: \"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c\") " Oct 30 23:41:22.395002 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:22.394969 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-openshift-ca" pod "e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c" (UID: "e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:41:22.395390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:22.395357 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:22.396802 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:22.396776 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-xks-ca" pod "e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c" (UID: "e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:41:22.396904 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:22.396874 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-kube-api-access-wcxp2" pod "e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c" (UID: "e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c"). InnerVolumeSpecName "kube-api-access-wcxp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:41:22.497981 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:22.497943 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcxp2\" (UniqueName: \"kubernetes.io/projected/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-kube-api-access-wcxp2\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:22.497981 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:22.497982 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:23.086370 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:23.086332 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" Oct 30 23:41:23.086370 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:23.086365 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp" event={"ID":"e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c","Type":"ContainerDied","Data":"a668e892f0dd78a1b6993151fbb5ea62e77f4037df7be65f36eb1cf30fa3977f"} Oct 30 23:41:23.086648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:23.086416 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a668e892f0dd78a1b6993151fbb5ea62e77f4037df7be65f36eb1cf30fa3977f" Oct 30 23:41:23.100124 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:23.100101 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:25.661572 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.661539 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724"] Oct 30 23:41:25.662199 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.662184 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c" containerName="revoke-keys" Oct 30 23:41:25.662246 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.662200 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c" containerName="revoke-keys" Oct 30 23:41:25.662348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.662339 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c" containerName="revoke-keys" Oct 30 23:41:25.666205 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.666189 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.686802 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.686765 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724"] Oct 30 23:41:25.743497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.743441 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.743725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.743563 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.743725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.743643 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2658215-b85d-44d6-8d22-9803dd36e9f4-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.743725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.743718 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sf6f\" (UniqueName: \"kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-kube-api-access-8sf6f\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.743877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.743807 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.743877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.743851 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.743969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.743897 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.744060 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.744038 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.744160 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.744114 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.845363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.845328 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.845545 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.845409 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.845545 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.845436 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.845545 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.845519 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2658215-b85d-44d6-8d22-9803dd36e9f4-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.845669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.845565 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sf6f\" (UniqueName: \"kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-kube-api-access-8sf6f\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.845669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.845652 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.845761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.845694 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.845761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.845740 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.845862 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.845835 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.845862 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.845853 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.845963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.845928 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.845963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.845936 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.846125 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.846103 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.846339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.846312 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2658215-b85d-44d6-8d22-9803dd36e9f4-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.848181 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.848163 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.848591 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.848565 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.858081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.858055 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.858319 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.858301 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sf6f\" (UniqueName: \"kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-kube-api-access-8sf6f\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:25.946127 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.946041 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28"] Oct 30 23:41:25.951740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.951718 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:25.977530 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.977497 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28"] Oct 30 23:41:25.981080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:25.981053 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:26.049082 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.049022 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.049526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.049118 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.049526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.049168 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.049526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.049202 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.049526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.049259 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d474j\" (UniqueName: \"kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-kube-api-access-d474j\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.049526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.049298 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.049526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.049484 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.050256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.049556 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.050256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.049611 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.095366 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.095321 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5"] Oct 30 23:41:26.107211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.106287 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.121254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.121213 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5"] Oct 30 23:41:26.150878 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.150842 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.151067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151000 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.151141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151070 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.151141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151115 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.151254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151170 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.151254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151213 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.151377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151280 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.151377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151325 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.151377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151361 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.151567 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151387 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfjp\" (UniqueName: \"kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-kube-api-access-bjfjp\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.151567 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151424 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.151567 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151436 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.151567 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151551 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.151766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151596 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.151766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151670 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d474j\" (UniqueName: \"kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-kube-api-access-d474j\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.151766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151715 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.151766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151750 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.151964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151790 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.151964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.151836 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.152304 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.152280 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.152736 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.152707 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.152884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.152865 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.152979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.152958 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.154868 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.154844 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.155185 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.155163 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.164655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.164636 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.164827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.164805 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d474j\" (UniqueName: \"kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-kube-api-access-d474j\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.252868 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.252789 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.252868 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.252820 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfjp\" (UniqueName: \"kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-kube-api-access-bjfjp\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.253072 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.252898 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.253072 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.252920 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.253072 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.252960 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.253072 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.253024 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.253304 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.253136 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.253304 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.253218 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.253304 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.253292 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.253543 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.253522 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.253607 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.253582 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.253607 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.253591 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.253739 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.253717 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.253833 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.253814 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.255588 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.255566 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.255766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.255747 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.267841 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.267811 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:26.275185 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.275160 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfjp\" (UniqueName: \"kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-kube-api-access-bjfjp\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.275760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.275744 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.301662 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:41:26.301625 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2658215_b85d_44d6_8d22_9803dd36e9f4.slice/crio-7236e5d7c15f1aef222038b0f63cca98fac7bba5b2ee6e4726eea0d330d588ef WatchSource:0}: Error finding container 7236e5d7c15f1aef222038b0f63cca98fac7bba5b2ee6e4726eea0d330d588ef: Status 404 returned error can't find the container with id 7236e5d7c15f1aef222038b0f63cca98fac7bba5b2ee6e4726eea0d330d588ef Oct 30 23:41:26.302353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.302330 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724"] Oct 30 23:41:26.304595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.304558 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:26.304669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.304631 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:26.304722 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.304667 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:26.433515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.432436 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:26.573720 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:41:26.571595 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbbfae7_a8ac_4546_b3e9_3867ce25cfeb.slice/crio-6b366f947309bc8d04548ad5f5d8fe848b3cb8ee38c9e999e7159962820e7305 WatchSource:0}: Error finding container 6b366f947309bc8d04548ad5f5d8fe848b3cb8ee38c9e999e7159962820e7305: Status 404 returned error can't find the container with id 6b366f947309bc8d04548ad5f5d8fe848b3cb8ee38c9e999e7159962820e7305 Oct 30 23:41:26.576953 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.576925 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28"] Oct 30 23:41:26.578424 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.578389 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:26.578539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.578456 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:26.578539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.578525 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:26.727273 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.727242 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5"] Oct 30 23:41:26.727814 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:41:26.727732 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc946897_4a76_4a95_b7ed_c9402f9a4d42.slice/crio-4d41f91df84b22080743025a5491a58ef4008d9df24ec332c218cff2c5153ec8 WatchSource:0}: Error finding container 4d41f91df84b22080743025a5491a58ef4008d9df24ec332c218cff2c5153ec8: Status 404 returned error can't find the container with id 4d41f91df84b22080743025a5491a58ef4008d9df24ec332c218cff2c5153ec8 Oct 30 23:41:26.731040 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.731000 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:26.731129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.731089 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:26.731166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:26.731129 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:27.135591 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.135489 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" event={"ID":"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb","Type":"ContainerStarted","Data":"359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79"} Oct 30 23:41:27.135761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.135596 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" event={"ID":"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb","Type":"ContainerStarted","Data":"6b366f947309bc8d04548ad5f5d8fe848b3cb8ee38c9e999e7159962820e7305"} Oct 30 23:41:27.137432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.137408 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" event={"ID":"bc946897-4a76-4a95-b7ed-c9402f9a4d42","Type":"ContainerStarted","Data":"dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272"} Oct 30 23:41:27.137604 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.137495 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" event={"ID":"bc946897-4a76-4a95-b7ed-c9402f9a4d42","Type":"ContainerStarted","Data":"4d41f91df84b22080743025a5491a58ef4008d9df24ec332c218cff2c5153ec8"} Oct 30 23:41:27.139349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.139327 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" event={"ID":"d2658215-b85d-44d6-8d22-9803dd36e9f4","Type":"ContainerStarted","Data":"47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e"} Oct 30 23:41:27.139434 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.139365 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" event={"ID":"d2658215-b85d-44d6-8d22-9803dd36e9f4","Type":"ContainerStarted","Data":"7236e5d7c15f1aef222038b0f63cca98fac7bba5b2ee6e4726eea0d330d588ef"} Oct 30 23:41:27.253042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.252957 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podStartSLOduration=2.252942442 podStartE2EDuration="2.252942442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:27.249846545 +0000 UTC m=+1298.768847636" watchObservedRunningTime="2026-10-30 23:41:27.252942442 +0000 UTC m=+1298.771943595" Oct 30 23:41:27.268560 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.268528 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:27.269974 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.269947 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 23:41:27.270130 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.270002 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 23:41:27.318445 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.318389 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" podStartSLOduration=1.318371036 podStartE2EDuration="1.318371036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:27.298871904 +0000 UTC m=+1298.817872996" watchObservedRunningTime="2026-10-30 23:41:27.318371036 +0000 UTC m=+1298.837372125" Oct 30 23:41:27.437018 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.436927 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:27.439637 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.439603 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:41:27.439865 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.439839 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 23:41:27.981806 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.981773 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:27.991086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.990697 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:41:27.991086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:27.990756 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 23:41:28.275575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:28.269853 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 23:41:28.275575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:28.269934 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 23:41:28.436491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:28.435885 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:41:28.436491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:28.435949 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 23:41:28.982515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:28.982479 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:41:28.983008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:28.982553 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 23:41:29.274543 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:29.274222 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 23:41:29.275494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:29.274623 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 23:41:29.434529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:29.432833 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:41:29.434529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:29.432897 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 23:41:29.608495 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:41:29.596806 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:41:29.982413 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:29.982330 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:41:29.982413 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:29.982383 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 23:41:30.271455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:30.268685 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 23:41:30.271455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:30.268766 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 23:41:30.458654 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:30.458611 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:41:30.458957 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:30.458925 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 23:41:30.990490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:30.989780 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:41:30.990490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:30.989849 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 23:41:31.282726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:31.277062 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 23:41:31.282726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:31.277135 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 23:41:31.436492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:31.435582 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:41:31.436492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:31.435673 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 23:41:31.648255 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:31.648207 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:41:31.691603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:31.691554 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:31.805640 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:31.805591 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:31.912842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:31.912734 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:32.000490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:31.997666 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:41:32.000490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:31.997728 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 23:41:32.034484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.033606 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:32.067604 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.067551 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:41:32.093390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.093347 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:32.123114 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.123071 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:41:32.137194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.137153 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:32.231760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.231658 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:32.272488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.271878 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 23:41:32.272488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.271943 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 23:41:32.342488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.341549 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:32.444490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.433833 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:41:32.444490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.433896 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 23:41:32.659081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.659035 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:32.663061 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.659348 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:32.663648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.663629 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:41:32.715487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.702822 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:32.715487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.703549 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:41:32.761490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.738557 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:32.762098 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.759498 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podStartSLOduration=7.759475244 podStartE2EDuration="7.759475244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:27.340888011 +0000 UTC m=+1298.859889103" watchObservedRunningTime="2026-10-30 23:41:32.759475244 +0000 UTC m=+1304.278476326" Oct 30 23:41:32.807657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.807616 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:32.853528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.837213 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:41:32.913775 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.913683 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:32.917483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:32.914305 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:41:33.036443 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.034542 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:33.036443 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.034846 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:41:33.139681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.128019 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:41:33.139681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.128083 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 23:41:33.139681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.130299 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:33.139681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.139586 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:41:33.143685 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.140212 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:41:33.143685 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.140563 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:41:33.143685 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.140696 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:33.143685 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.140909 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:41:33.164546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.152606 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:33.164546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.152669 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:41:33.266127 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.235778 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:33.266665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.266639 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:41:33.285623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.285574 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:33.431681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.431571 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:33.450689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.450654 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:41:33.508487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.496833 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 23:41:33.508487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.496899 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 23:41:33.508487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.497208 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:41:33.508487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.497245 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 23:41:33.663473 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.663363 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:33.840665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.834364 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:33.840665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.835173 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:41:33.840665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.835322 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:41:33.840665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.838582 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:33.958488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:33.954820 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:41:34.114885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:34.022667 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:41:34.114885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:34.022749 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 23:41:34.114885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:34.041985 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:34.154689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:34.132418 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:34.155027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:34.154501 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:41:34.155994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:34.154572 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:34.155994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:34.154595 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:41:34.281485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:34.274141 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:34.874531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:34.309699 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 23:41:34.874531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:34.861561 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 23:41:34.938983 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:34.938926 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:41:35.026580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.000004 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:35.026580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.024380 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:41:35.081851 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.060901 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:41:35.117406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.117339 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:35.261224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.188260 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:41:35.262054 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.189099 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:41:35.262274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.262243 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 23:41:35.262383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.240553 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:41:35.281486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.240019 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:35.350791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.328670 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:41:35.350791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.331768 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:41:35.350791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.331817 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 23:41:35.447367 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.447326 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:41:35.455813 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.455783 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:41:35.456550 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.456525 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:35.459723 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.459693 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:41:35.459880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.459860 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 23:41:35.466514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.460737 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 23:41:35.466701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.466678 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 23:41:35.490483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.486847 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:41:35.490483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.486915 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" podUID="3d62150f-cb32-4975-a88e-70b671493882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:35.771278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.770974 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:41:35.771707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.771692 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:41:35.773525 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.773217 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:41:35.774308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.774294 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:41:35.775995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.775978 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:41:35.776166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.776154 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:41:35.776875 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.776860 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:41:35.777631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.777616 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:41:35.778348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.778334 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:41:35.779074 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.779059 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:41:35.779563 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.779539 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:35.779997 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:35.779973 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:41:36.321591 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:36.321554 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 23:41:36.322183 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:36.322149 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 23:41:36.433555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:36.433063 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:41:36.433555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:36.433133 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 23:41:36.998186 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:36.997730 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:41:36.998186 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:36.997811 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:37.271226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:37.269809 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:37.286608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:37.277980 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:37.342654 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:37.342627 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:41:37.602947 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:37.602916 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:37.604853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:37.604831 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:37.605867 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:37.605850 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:37.606750 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:37.606622 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:37.609361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:37.608391 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:41:37.615567 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:37.615496 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:41:38.010994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.010903 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq"] Oct 30 23:41:38.016514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.016471 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:38.022778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.022254 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:41:38.022778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.022327 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:41:38.028568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.028539 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq"] Oct 30 23:41:38.146934 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.146900 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82cd6a0d-t4mrq\" (UID: \"440b4040-ef5a-41e3-a505-7ca11b054f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:38.147224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.147199 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82cd6a0d-t4mrq\" (UID: \"440b4040-ef5a-41e3-a505-7ca11b054f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:38.147448 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.147431 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xt8h\" (UniqueName: \"kubernetes.io/projected/440b4040-ef5a-41e3-a505-7ca11b054f05-kube-api-access-8xt8h\") pod \"maas-api-revoke-sub-82cd6a0d-t4mrq\" (UID: \"440b4040-ef5a-41e3-a505-7ca11b054f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:38.248578 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.248546 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xt8h\" (UniqueName: \"kubernetes.io/projected/440b4040-ef5a-41e3-a505-7ca11b054f05-kube-api-access-8xt8h\") pod \"maas-api-revoke-sub-82cd6a0d-t4mrq\" (UID: \"440b4040-ef5a-41e3-a505-7ca11b054f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:38.248739 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.248618 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82cd6a0d-t4mrq\" (UID: \"440b4040-ef5a-41e3-a505-7ca11b054f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:38.248796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.248743 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82cd6a0d-t4mrq\" (UID: \"440b4040-ef5a-41e3-a505-7ca11b054f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:38.249310 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.249285 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82cd6a0d-t4mrq\" (UID: \"440b4040-ef5a-41e3-a505-7ca11b054f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:38.251814 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.251783 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82cd6a0d-t4mrq\" (UID: \"440b4040-ef5a-41e3-a505-7ca11b054f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:38.261226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.261160 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xt8h\" (UniqueName: \"kubernetes.io/projected/440b4040-ef5a-41e3-a505-7ca11b054f05-kube-api-access-8xt8h\") pod \"maas-api-revoke-sub-82cd6a0d-t4mrq\" (UID: \"440b4040-ef5a-41e3-a505-7ca11b054f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:38.349517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.349477 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:38.641576 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:41:38.641544 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440b4040_ef5a_41e3_a505_7ca11b054f05.slice/crio-ca56ef883b88030c96a3348aa135788ec95d40b04bd8e8a8ce5a2e7a6cab61cb WatchSource:0}: Error finding container ca56ef883b88030c96a3348aa135788ec95d40b04bd8e8a8ce5a2e7a6cab61cb: Status 404 returned error can't find the container with id ca56ef883b88030c96a3348aa135788ec95d40b04bd8e8a8ce5a2e7a6cab61cb Oct 30 23:41:38.642541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.642512 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq"] Oct 30 23:41:38.774738 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:38.774652 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" event={"ID":"440b4040-ef5a-41e3-a505-7ca11b054f05","Type":"ContainerStarted","Data":"ca56ef883b88030c96a3348aa135788ec95d40b04bd8e8a8ce5a2e7a6cab61cb"} Oct 30 23:41:39.541579 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:39.541447 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9"] Oct 30 23:41:39.542198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:39.541699 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" podUID="8f37a547-8183-4386-9f68-d4841e3a252a" containerName="istio-proxy" containerID="cri-o://b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035" gracePeriod=30 Oct 30 23:41:39.783401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:39.783358 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" event={"ID":"440b4040-ef5a-41e3-a505-7ca11b054f05","Type":"ContainerStarted","Data":"ccfaa9afd03f2b3057283ed590fb9cedcbdf99f62068007c88a5f703be073dbe"} Oct 30 23:41:39.802655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:39.802599 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" podStartSLOduration=2.802582591 podStartE2EDuration="2.802582591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:39.801206346 +0000 UTC m=+1311.320207438" watchObservedRunningTime="2026-10-30 23:41:39.802582591 +0000 UTC m=+1311.321583685" Oct 30 23:41:40.746365 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.746334 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v"] Oct 30 23:41:40.751379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.751356 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:40.766244 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.766214 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v"] Oct 30 23:41:40.808212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.808169 7570 generic.go:356] "Generic (PLEG): container finished" podID="440b4040-ef5a-41e3-a505-7ca11b054f05" containerID="ccfaa9afd03f2b3057283ed590fb9cedcbdf99f62068007c88a5f703be073dbe" exitCode=0 Oct 30 23:41:40.808412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.808251 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" event={"ID":"440b4040-ef5a-41e3-a505-7ca11b054f05","Type":"ContainerDied","Data":"ccfaa9afd03f2b3057283ed590fb9cedcbdf99f62068007c88a5f703be073dbe"} Oct 30 23:41:40.818498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.818435 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbv7\" (UniqueName: \"kubernetes.io/projected/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-kube-api-access-cwbv7\") pod \"maas-api-revoke-sub-1572848f-bbx4v\" (UID: \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:40.818624 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.818558 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1572848f-bbx4v\" (UID: \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:40.818624 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.818592 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1572848f-bbx4v\" (UID: \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:40.919945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.919907 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbv7\" (UniqueName: \"kubernetes.io/projected/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-kube-api-access-cwbv7\") pod \"maas-api-revoke-sub-1572848f-bbx4v\" (UID: \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:40.920094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.919963 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1572848f-bbx4v\" (UID: \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:40.920094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.919996 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1572848f-bbx4v\" (UID: \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:40.920784 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.920723 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1572848f-bbx4v\" (UID: \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:40.922377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.922353 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1572848f-bbx4v\" (UID: \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:40.932379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:40.932354 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbv7\" (UniqueName: \"kubernetes.io/projected/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-kube-api-access-cwbv7\") pod \"maas-api-revoke-sub-1572848f-bbx4v\" (UID: \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:41.067581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:41.067493 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:41.352969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:41.352941 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v"] Oct 30 23:41:41.353745 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:41:41.353720 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca3a700_3162_4b5a_861f_69b6bee3d3e9.slice/crio-b6fb185ecd82c4d089b2e5a6046d6161dc3473909f982cff0be25e12fcfc20f9 WatchSource:0}: Error finding container b6fb185ecd82c4d089b2e5a6046d6161dc3473909f982cff0be25e12fcfc20f9: Status 404 returned error can't find the container with id b6fb185ecd82c4d089b2e5a6046d6161dc3473909f982cff0be25e12fcfc20f9 Oct 30 23:41:41.817715 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:41.817668 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" event={"ID":"7ca3a700-3162-4b5a-861f-69b6bee3d3e9","Type":"ContainerStarted","Data":"eb0dba80b3ea9d8ef87bc91ff93a6a9814ba07075c2fc2be805ec37e5ff0a4e8"} Oct 30 23:41:41.818164 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:41.817730 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" event={"ID":"7ca3a700-3162-4b5a-861f-69b6bee3d3e9","Type":"ContainerStarted","Data":"b6fb185ecd82c4d089b2e5a6046d6161dc3473909f982cff0be25e12fcfc20f9"} Oct 30 23:41:41.838681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:41.838632 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" podStartSLOduration=1.8386166940000002 podStartE2EDuration="1.838616694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:41.836806368 +0000 UTC m=+1313.355807459" watchObservedRunningTime="2026-10-30 23:41:41.838616694 +0000 UTC m=+1313.357617784" Oct 30 23:41:42.035904 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.035878 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:42.041831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.041809 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440b4040-ef5a-41e3-a505-7ca11b054f05-kube-api-access-8xt8h\" (UniqueName: \"kubernetes.io/projected/440b4040-ef5a-41e3-a505-7ca11b054f05-kube-api-access-8xt8h\") pod \"440b4040-ef5a-41e3-a505-7ca11b054f05\" (UID: \"440b4040-ef5a-41e3-a505-7ca11b054f05\") " Oct 30 23:41:42.042085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.042067 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-xks-ca\") pod \"440b4040-ef5a-41e3-a505-7ca11b054f05\" (UID: \"440b4040-ef5a-41e3-a505-7ca11b054f05\") " Oct 30 23:41:42.042210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.042193 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-openshift-ca\") pod \"440b4040-ef5a-41e3-a505-7ca11b054f05\" (UID: \"440b4040-ef5a-41e3-a505-7ca11b054f05\") " Oct 30 23:41:42.042604 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.042568 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-openshift-ca" pod "440b4040-ef5a-41e3-a505-7ca11b054f05" (UID: "440b4040-ef5a-41e3-a505-7ca11b054f05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:41:42.042954 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.042932 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:42.044167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.044137 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-xks-ca" pod "440b4040-ef5a-41e3-a505-7ca11b054f05" (UID: "440b4040-ef5a-41e3-a505-7ca11b054f05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:41:42.044286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.044199 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440b4040-ef5a-41e3-a505-7ca11b054f05-kube-api-access-8xt8h" pod "440b4040-ef5a-41e3-a505-7ca11b054f05" (UID: "440b4040-ef5a-41e3-a505-7ca11b054f05"). InnerVolumeSpecName "kube-api-access-8xt8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:41:42.144209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.144172 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440b4040-ef5a-41e3-a505-7ca11b054f05-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:42.144209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.144206 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xt8h\" (UniqueName: \"kubernetes.io/projected/440b4040-ef5a-41e3-a505-7ca11b054f05-kube-api-access-8xt8h\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:42.826827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.826794 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" Oct 30 23:41:42.827280 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.826802 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq" event={"ID":"440b4040-ef5a-41e3-a505-7ca11b054f05","Type":"ContainerDied","Data":"ca56ef883b88030c96a3348aa135788ec95d40b04bd8e8a8ce5a2e7a6cab61cb"} Oct 30 23:41:42.827280 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.826920 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca56ef883b88030c96a3348aa135788ec95d40b04bd8e8a8ce5a2e7a6cab61cb" Oct 30 23:41:42.828541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.828516 7570 generic.go:356] "Generic (PLEG): container finished" podID="7ca3a700-3162-4b5a-861f-69b6bee3d3e9" containerID="eb0dba80b3ea9d8ef87bc91ff93a6a9814ba07075c2fc2be805ec37e5ff0a4e8" exitCode=0 Oct 30 23:41:42.828672 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:42.828605 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" event={"ID":"7ca3a700-3162-4b5a-861f-69b6bee3d3e9","Type":"ContainerDied","Data":"eb0dba80b3ea9d8ef87bc91ff93a6a9814ba07075c2fc2be805ec37e5ff0a4e8"} Oct 30 23:41:43.899296 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:43.899252 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb"] Oct 30 23:41:43.901834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:43.901811 7570 cpu_manager.go:414] "containerMap: removing container" podUID="440b4040-ef5a-41e3-a505-7ca11b054f05" containerName="revoke-keys" Oct 30 23:41:43.901979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:43.901964 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440b4040-ef5a-41e3-a505-7ca11b054f05" containerName="revoke-keys" Oct 30 23:41:43.902549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:43.902441 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="440b4040-ef5a-41e3-a505-7ca11b054f05" containerName="revoke-keys" Oct 30 23:41:43.920492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:43.920422 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb"] Oct 30 23:41:43.920700 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:43.920581 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:43.979487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:43.979438 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0074fabf-xc9fb\" (UID: \"85070cda-919f-4fd5-ad70-d50b6852248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:43.979658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:43.979548 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqcbp\" (UniqueName: \"kubernetes.io/projected/85070cda-919f-4fd5-ad70-d50b6852248c-kube-api-access-kqcbp\") pod \"maas-api-revoke-sub-0074fabf-xc9fb\" (UID: \"85070cda-919f-4fd5-ad70-d50b6852248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:43.979658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:43.979581 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0074fabf-xc9fb\" (UID: \"85070cda-919f-4fd5-ad70-d50b6852248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:44.037519 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.037495 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:44.081221 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.081190 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-openshift-ca\") pod \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\" (UID: \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\") " Oct 30 23:41:44.081491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.081442 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-kube-api-access-cwbv7\" (UniqueName: \"kubernetes.io/projected/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-kube-api-access-cwbv7\") pod \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\" (UID: \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\") " Oct 30 23:41:44.081576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.081532 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-openshift-ca" pod "7ca3a700-3162-4b5a-861f-69b6bee3d3e9" (UID: "7ca3a700-3162-4b5a-861f-69b6bee3d3e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:41:44.081576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.081564 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-xks-ca\") pod \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\" (UID: \"7ca3a700-3162-4b5a-861f-69b6bee3d3e9\") " Oct 30 23:41:44.081951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.081931 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0074fabf-xc9fb\" (UID: \"85070cda-919f-4fd5-ad70-d50b6852248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:44.082030 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.081996 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqcbp\" (UniqueName: \"kubernetes.io/projected/85070cda-919f-4fd5-ad70-d50b6852248c-kube-api-access-kqcbp\") pod \"maas-api-revoke-sub-0074fabf-xc9fb\" (UID: \"85070cda-919f-4fd5-ad70-d50b6852248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:44.082087 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.082033 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0074fabf-xc9fb\" (UID: \"85070cda-919f-4fd5-ad70-d50b6852248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:44.082506 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.082431 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:44.082799 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.082774 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0074fabf-xc9fb\" (UID: \"85070cda-919f-4fd5-ad70-d50b6852248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:44.083839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.083812 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-xks-ca" pod "7ca3a700-3162-4b5a-861f-69b6bee3d3e9" (UID: "7ca3a700-3162-4b5a-861f-69b6bee3d3e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:41:44.083953 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.083882 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-kube-api-access-cwbv7" pod "7ca3a700-3162-4b5a-861f-69b6bee3d3e9" (UID: "7ca3a700-3162-4b5a-861f-69b6bee3d3e9"). InnerVolumeSpecName "kube-api-access-cwbv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:41:44.084653 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.084625 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0074fabf-xc9fb\" (UID: \"85070cda-919f-4fd5-ad70-d50b6852248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:44.094869 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.094818 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqcbp\" (UniqueName: \"kubernetes.io/projected/85070cda-919f-4fd5-ad70-d50b6852248c-kube-api-access-kqcbp\") pod \"maas-api-revoke-sub-0074fabf-xc9fb\" (UID: \"85070cda-919f-4fd5-ad70-d50b6852248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:44.184276 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.184236 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwbv7\" (UniqueName: \"kubernetes.io/projected/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-kube-api-access-cwbv7\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:44.184276 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.184270 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ca3a700-3162-4b5a-861f-69b6bee3d3e9-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:44.241517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.241451 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:44.533833 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.533802 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb"] Oct 30 23:41:44.534620 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:41:44.534591 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85070cda_919f_4fd5_ad70_d50b6852248c.slice/crio-99e6ea4019afef94910509727e1e6c3a376e8efd18a923e7759bdeb1fda5d9fa WatchSource:0}: Error finding container 99e6ea4019afef94910509727e1e6c3a376e8efd18a923e7759bdeb1fda5d9fa: Status 404 returned error can't find the container with id 99e6ea4019afef94910509727e1e6c3a376e8efd18a923e7759bdeb1fda5d9fa Oct 30 23:41:44.827311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.827283 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:41:44.844517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.844485 7570 generic.go:356] "Generic (PLEG): container finished" podID="8f37a547-8183-4386-9f68-d4841e3a252a" containerID="b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035" exitCode=0 Oct 30 23:41:44.844670 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.844521 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" event={"ID":"8f37a547-8183-4386-9f68-d4841e3a252a","Type":"ContainerDied","Data":"b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035"} Oct 30 23:41:44.844670 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.844557 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" Oct 30 23:41:44.844670 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.844589 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9" event={"ID":"8f37a547-8183-4386-9f68-d4841e3a252a","Type":"ContainerDied","Data":"4a688d8e7ce9db68252aa18b9cd3a8c21bf8fea077fdd6e4edff4120181d214f"} Oct 30 23:41:44.844670 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.844623 7570 scope.go:122] "RemoveContainer" containerID="b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035" Oct 30 23:41:44.846652 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.846626 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" Oct 30 23:41:44.846761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.846675 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v" event={"ID":"7ca3a700-3162-4b5a-861f-69b6bee3d3e9","Type":"ContainerDied","Data":"b6fb185ecd82c4d089b2e5a6046d6161dc3473909f982cff0be25e12fcfc20f9"} Oct 30 23:41:44.846761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.846737 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6fb185ecd82c4d089b2e5a6046d6161dc3473909f982cff0be25e12fcfc20f9" Oct 30 23:41:44.848311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.848274 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" event={"ID":"85070cda-919f-4fd5-ad70-d50b6852248c","Type":"ContainerStarted","Data":"99e6ea4019afef94910509727e1e6c3a376e8efd18a923e7759bdeb1fda5d9fa"} Oct 30 23:41:44.858774 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.857780 7570 scope.go:122] "RemoveContainer" containerID="b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035" Oct 30 23:41:44.858774 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:41:44.858099 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035\": container with ID starting with b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035 not found: ID does not exist" containerID="b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035" Oct 30 23:41:44.858774 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.858129 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035"} err="failed to get container status \"b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035\": rpc error: code = NotFound desc = could not find container \"b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035\": container with ID starting with b03aa18246904cba14c44ffc3f4490aff8f1f374ec78ec3957a62ea724551035 not found: ID does not exist" Oct 30 23:41:44.894275 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.894249 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-kube-api-access-f99dr\" (UniqueName: \"kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-kube-api-access-f99dr\") pod \"8f37a547-8183-4386-9f68-d4841e3a252a\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " Oct 30 23:41:44.894410 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.894327 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-credential-socket\") pod \"8f37a547-8183-4386-9f68-d4841e3a252a\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " Oct 30 23:41:44.894410 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.894361 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-data\") pod \"8f37a547-8183-4386-9f68-d4841e3a252a\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " Oct 30 23:41:44.894566 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.894395 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-socket\") pod \"8f37a547-8183-4386-9f68-d4841e3a252a\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " Oct 30 23:41:44.894566 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.894472 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f37a547-8183-4386-9f68-d4841e3a252a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f37a547-8183-4386-9f68-d4841e3a252a-istio-podinfo\") pod \"8f37a547-8183-4386-9f68-d4841e3a252a\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " Oct 30 23:41:44.894700 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.894632 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-certs\") pod \"8f37a547-8183-4386-9f68-d4841e3a252a\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " Oct 30 23:41:44.894700 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.894687 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-istio-token\" (UniqueName: \"kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-istio-token\") pod \"8f37a547-8183-4386-9f68-d4841e3a252a\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " Oct 30 23:41:44.894796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.894694 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-data" pod "8f37a547-8183-4386-9f68-d4841e3a252a" (UID: "8f37a547-8183-4386-9f68-d4841e3a252a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:41:44.894796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.894743 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-socket" pod "8f37a547-8183-4386-9f68-d4841e3a252a" (UID: "8f37a547-8183-4386-9f68-d4841e3a252a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:41:44.894899 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.894840 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-envoy\") pod \"8f37a547-8183-4386-9f68-d4841e3a252a\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " Oct 30 23:41:44.894951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.894936 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f37a547-8183-4386-9f68-d4841e3a252a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f37a547-8183-4386-9f68-d4841e3a252a-istiod-ca-cert\") pod \"8f37a547-8183-4386-9f68-d4841e3a252a\" (UID: \"8f37a547-8183-4386-9f68-d4841e3a252a\") " Oct 30 23:41:44.895659 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.895636 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:44.895791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.895656 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-credential-socket" pod "8f37a547-8183-4386-9f68-d4841e3a252a" (UID: "8f37a547-8183-4386-9f68-d4841e3a252a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:41:44.895791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.895663 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:44.895949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.895924 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f37a547-8183-4386-9f68-d4841e3a252a-istiod-ca-cert" pod "8f37a547-8183-4386-9f68-d4841e3a252a" (UID: "8f37a547-8183-4386-9f68-d4841e3a252a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:41:44.896089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.896060 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-certs" pod "8f37a547-8183-4386-9f68-d4841e3a252a" (UID: "8f37a547-8183-4386-9f68-d4841e3a252a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:41:44.897303 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.897275 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-kube-api-access-f99dr" pod "8f37a547-8183-4386-9f68-d4841e3a252a" (UID: "8f37a547-8183-4386-9f68-d4841e3a252a"). InnerVolumeSpecName "kube-api-access-f99dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:41:44.898378 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.898346 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-envoy" pod "8f37a547-8183-4386-9f68-d4841e3a252a" (UID: "8f37a547-8183-4386-9f68-d4841e3a252a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:41:44.899171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.899145 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f37a547-8183-4386-9f68-d4841e3a252a-istio-podinfo" pod "8f37a547-8183-4386-9f68-d4841e3a252a" (UID: "8f37a547-8183-4386-9f68-d4841e3a252a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:41:44.899256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.899232 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-istio-token" pod "8f37a547-8183-4386-9f68-d4841e3a252a" (UID: "8f37a547-8183-4386-9f68-d4841e3a252a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:41:44.996884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.996849 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:44.996884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.996880 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f37a547-8183-4386-9f68-d4841e3a252a-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:44.997379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.996895 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f99dr\" (UniqueName: \"kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-kube-api-access-f99dr\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:44.997379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.996908 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:44.997379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.996920 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f37a547-8183-4386-9f68-d4841e3a252a-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:44.997379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.996932 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f37a547-8183-4386-9f68-d4841e3a252a-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:44.997379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:44.996945 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f37a547-8183-4386-9f68-d4841e3a252a-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:45.171425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:45.171390 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9"] Oct 30 23:41:45.175192 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:45.175163 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r5qv9"] Oct 30 23:41:45.858503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:45.858449 7570 generic.go:356] "Generic (PLEG): container finished" podID="85070cda-919f-4fd5-ad70-d50b6852248c" containerID="7c2e7bf7ac10359f0ff0fcd9dc24c198906da34288dc320d68ed9febcf54433e" exitCode=0 Oct 30 23:41:45.859023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:45.859002 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" event={"ID":"85070cda-919f-4fd5-ad70-d50b6852248c","Type":"ContainerDied","Data":"7c2e7bf7ac10359f0ff0fcd9dc24c198906da34288dc320d68ed9febcf54433e"} Oct 30 23:41:46.612436 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:46.608659 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f37a547-8183-4386-9f68-d4841e3a252a" path="/var/lib/kubelet/pods/8f37a547-8183-4386-9f68-d4841e3a252a/volumes" Oct 30 23:41:47.049180 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.049151 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:47.132353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.132255 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85070cda-919f-4fd5-ad70-d50b6852248c-kube-api-access-kqcbp\" (UniqueName: \"kubernetes.io/projected/85070cda-919f-4fd5-ad70-d50b6852248c-kube-api-access-kqcbp\") pod \"85070cda-919f-4fd5-ad70-d50b6852248c\" (UID: \"85070cda-919f-4fd5-ad70-d50b6852248c\") " Oct 30 23:41:47.132353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.132335 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-xks-ca\") pod \"85070cda-919f-4fd5-ad70-d50b6852248c\" (UID: \"85070cda-919f-4fd5-ad70-d50b6852248c\") " Oct 30 23:41:47.132627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.132423 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-openshift-ca\") pod \"85070cda-919f-4fd5-ad70-d50b6852248c\" (UID: \"85070cda-919f-4fd5-ad70-d50b6852248c\") " Oct 30 23:41:47.133351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.133322 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-openshift-ca" pod "85070cda-919f-4fd5-ad70-d50b6852248c" (UID: "85070cda-919f-4fd5-ad70-d50b6852248c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:41:47.135583 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.135554 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85070cda-919f-4fd5-ad70-d50b6852248c-kube-api-access-kqcbp" pod "85070cda-919f-4fd5-ad70-d50b6852248c" (UID: "85070cda-919f-4fd5-ad70-d50b6852248c"). InnerVolumeSpecName "kube-api-access-kqcbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:41:47.135914 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.135873 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-xks-ca" pod "85070cda-919f-4fd5-ad70-d50b6852248c" (UID: "85070cda-919f-4fd5-ad70-d50b6852248c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:41:47.234385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.234347 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqcbp\" (UniqueName: \"kubernetes.io/projected/85070cda-919f-4fd5-ad70-d50b6852248c-kube-api-access-kqcbp\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:47.234385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.234382 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:47.234385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.234394 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85070cda-919f-4fd5-ad70-d50b6852248c-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:47.877528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.877499 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" Oct 30 23:41:47.877909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.877497 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb" event={"ID":"85070cda-919f-4fd5-ad70-d50b6852248c","Type":"ContainerDied","Data":"99e6ea4019afef94910509727e1e6c3a376e8efd18a923e7759bdeb1fda5d9fa"} Oct 30 23:41:47.877909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:47.877616 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99e6ea4019afef94910509727e1e6c3a376e8efd18a923e7759bdeb1fda5d9fa" Oct 30 23:41:50.985940 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.985909 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd"] Oct 30 23:41:50.986729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.986711 7570 cpu_manager.go:414] "containerMap: removing container" podUID="7ca3a700-3162-4b5a-861f-69b6bee3d3e9" containerName="revoke-keys" Oct 30 23:41:50.986729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.986730 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca3a700-3162-4b5a-861f-69b6bee3d3e9" containerName="revoke-keys" Oct 30 23:41:50.986857 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.986741 7570 cpu_manager.go:414] "containerMap: removing container" podUID="8f37a547-8183-4386-9f68-d4841e3a252a" containerName="istio-proxy" Oct 30 23:41:50.986857 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.986745 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f37a547-8183-4386-9f68-d4841e3a252a" containerName="istio-proxy" Oct 30 23:41:50.986857 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.986767 7570 cpu_manager.go:414] "containerMap: removing container" podUID="85070cda-919f-4fd5-ad70-d50b6852248c" containerName="revoke-keys" Oct 30 23:41:50.986857 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.986773 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85070cda-919f-4fd5-ad70-d50b6852248c" containerName="revoke-keys" Oct 30 23:41:50.987010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.986874 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca3a700-3162-4b5a-861f-69b6bee3d3e9" containerName="revoke-keys" Oct 30 23:41:50.987010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.986884 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f37a547-8183-4386-9f68-d4841e3a252a" containerName="istio-proxy" Oct 30 23:41:50.987010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.986893 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="85070cda-919f-4fd5-ad70-d50b6852248c" containerName="revoke-keys" Oct 30 23:41:50.992204 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.992185 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:50.997451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.997428 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:41:50.998804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:50.998773 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:41:51.009298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.009270 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd"] Oct 30 23:41:51.094022 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.093988 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2k92\" (UniqueName: \"kubernetes.io/projected/2686bb47-640a-4dc2-81fb-945e17f98906-kube-api-access-k2k92\") pod \"maas-api-revoke-sub-8889cf6f-tngpd\" (UID: \"2686bb47-640a-4dc2-81fb-945e17f98906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:51.094209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.094105 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8889cf6f-tngpd\" (UID: \"2686bb47-640a-4dc2-81fb-945e17f98906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:51.094209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.094165 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8889cf6f-tngpd\" (UID: \"2686bb47-640a-4dc2-81fb-945e17f98906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:51.195552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.195511 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2k92\" (UniqueName: \"kubernetes.io/projected/2686bb47-640a-4dc2-81fb-945e17f98906-kube-api-access-k2k92\") pod \"maas-api-revoke-sub-8889cf6f-tngpd\" (UID: \"2686bb47-640a-4dc2-81fb-945e17f98906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:51.196109 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.196078 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8889cf6f-tngpd\" (UID: \"2686bb47-640a-4dc2-81fb-945e17f98906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:51.204648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.196171 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8889cf6f-tngpd\" (UID: \"2686bb47-640a-4dc2-81fb-945e17f98906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:51.204648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.197786 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8889cf6f-tngpd\" (UID: \"2686bb47-640a-4dc2-81fb-945e17f98906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:51.219781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.219687 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2k92\" (UniqueName: \"kubernetes.io/projected/2686bb47-640a-4dc2-81fb-945e17f98906-kube-api-access-k2k92\") pod \"maas-api-revoke-sub-8889cf6f-tngpd\" (UID: \"2686bb47-640a-4dc2-81fb-945e17f98906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:51.223119 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.223088 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8889cf6f-tngpd\" (UID: \"2686bb47-640a-4dc2-81fb-945e17f98906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:51.310025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.309947 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:51.594577 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:41:51.594543 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2686bb47_640a_4dc2_81fb_945e17f98906.slice/crio-f3d849704a3ee4369e6765a43a0b1584e5eccaf6ca9c44f1f4b69be0a35b49c3 WatchSource:0}: Error finding container f3d849704a3ee4369e6765a43a0b1584e5eccaf6ca9c44f1f4b69be0a35b49c3: Status 404 returned error can't find the container with id f3d849704a3ee4369e6765a43a0b1584e5eccaf6ca9c44f1f4b69be0a35b49c3 Oct 30 23:41:51.595172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.595147 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd"] Oct 30 23:41:51.909698 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:51.909615 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" event={"ID":"2686bb47-640a-4dc2-81fb-945e17f98906","Type":"ContainerStarted","Data":"f3d849704a3ee4369e6765a43a0b1584e5eccaf6ca9c44f1f4b69be0a35b49c3"} Oct 30 23:41:52.928335 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:52.928299 7570 generic.go:356] "Generic (PLEG): container finished" podID="2686bb47-640a-4dc2-81fb-945e17f98906" containerID="d6098a99f1a1102829afc800764271caa3b2afad2623cd9216ad27f435b35b51" exitCode=0 Oct 30 23:41:52.928800 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:52.928415 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" event={"ID":"2686bb47-640a-4dc2-81fb-945e17f98906","Type":"ContainerDied","Data":"d6098a99f1a1102829afc800764271caa3b2afad2623cd9216ad27f435b35b51"} Oct 30 23:41:54.102185 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.102161 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:54.220370 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.220272 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6"] Oct 30 23:41:54.221209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.221186 7570 cpu_manager.go:414] "containerMap: removing container" podUID="2686bb47-640a-4dc2-81fb-945e17f98906" containerName="revoke-keys" Oct 30 23:41:54.221209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.221209 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2686bb47-640a-4dc2-81fb-945e17f98906" containerName="revoke-keys" Oct 30 23:41:54.221504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.221489 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2686bb47-640a-4dc2-81fb-945e17f98906" containerName="revoke-keys" Oct 30 23:41:54.225828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.225811 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:41:54.244485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.234489 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6"] Oct 30 23:41:54.253402 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.244893 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-xks-ca\") pod \"2686bb47-640a-4dc2-81fb-945e17f98906\" (UID: \"2686bb47-640a-4dc2-81fb-945e17f98906\") " Oct 30 23:41:54.253402 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.246519 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-openshift-ca\") pod \"2686bb47-640a-4dc2-81fb-945e17f98906\" (UID: \"2686bb47-640a-4dc2-81fb-945e17f98906\") " Oct 30 23:41:54.253402 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.246663 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2686bb47-640a-4dc2-81fb-945e17f98906-kube-api-access-k2k92\" (UniqueName: \"kubernetes.io/projected/2686bb47-640a-4dc2-81fb-945e17f98906-kube-api-access-k2k92\") pod \"2686bb47-640a-4dc2-81fb-945e17f98906\" (UID: \"2686bb47-640a-4dc2-81fb-945e17f98906\") " Oct 30 23:41:54.270537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.270249 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-openshift-ca" pod "2686bb47-640a-4dc2-81fb-945e17f98906" (UID: "2686bb47-640a-4dc2-81fb-945e17f98906"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:41:54.271871 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.271844 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:54.285320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.285292 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-xks-ca" pod "2686bb47-640a-4dc2-81fb-945e17f98906" (UID: "2686bb47-640a-4dc2-81fb-945e17f98906"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:41:54.287207 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.287170 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2686bb47-640a-4dc2-81fb-945e17f98906-kube-api-access-k2k92" pod "2686bb47-640a-4dc2-81fb-945e17f98906" (UID: "2686bb47-640a-4dc2-81fb-945e17f98906"). InnerVolumeSpecName "kube-api-access-k2k92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:41:54.374577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.374544 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-441f67c6-xfmh6\" (UID: \"fd65f927-691b-412a-9dcb-2b79fca2213e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:41:54.374577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.374582 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pwwf\" (UniqueName: \"kubernetes.io/projected/fd65f927-691b-412a-9dcb-2b79fca2213e-kube-api-access-4pwwf\") pod \"maas-api-revoke-sub-441f67c6-xfmh6\" (UID: \"fd65f927-691b-412a-9dcb-2b79fca2213e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:41:54.374791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.374756 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-441f67c6-xfmh6\" (UID: \"fd65f927-691b-412a-9dcb-2b79fca2213e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:41:54.374841 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.374827 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2k92\" (UniqueName: \"kubernetes.io/projected/2686bb47-640a-4dc2-81fb-945e17f98906-kube-api-access-k2k92\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:54.374877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.374849 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2686bb47-640a-4dc2-81fb-945e17f98906-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:41:54.476190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.476103 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-441f67c6-xfmh6\" (UID: \"fd65f927-691b-412a-9dcb-2b79fca2213e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:41:54.476190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.476144 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwwf\" (UniqueName: \"kubernetes.io/projected/fd65f927-691b-412a-9dcb-2b79fca2213e-kube-api-access-4pwwf\") pod \"maas-api-revoke-sub-441f67c6-xfmh6\" (UID: \"fd65f927-691b-412a-9dcb-2b79fca2213e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:41:54.476386 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.476296 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-441f67c6-xfmh6\" (UID: \"fd65f927-691b-412a-9dcb-2b79fca2213e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:41:54.476891 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.476863 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-441f67c6-xfmh6\" (UID: \"fd65f927-691b-412a-9dcb-2b79fca2213e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:41:54.478892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.478870 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-441f67c6-xfmh6\" (UID: \"fd65f927-691b-412a-9dcb-2b79fca2213e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:41:54.493945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.493924 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwwf\" (UniqueName: \"kubernetes.io/projected/fd65f927-691b-412a-9dcb-2b79fca2213e-kube-api-access-4pwwf\") pod \"maas-api-revoke-sub-441f67c6-xfmh6\" (UID: \"fd65f927-691b-412a-9dcb-2b79fca2213e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:41:54.577013 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.576970 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:41:54.868006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.867968 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6"] Oct 30 23:41:54.872315 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:41:54.872279 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd65f927_691b_412a_9dcb_2b79fca2213e.slice/crio-838cf6619a9601c08935b24d353f1360ca765795b178131517ecad4a71025216 WatchSource:0}: Error finding container 838cf6619a9601c08935b24d353f1360ca765795b178131517ecad4a71025216: Status 404 returned error can't find the container with id 838cf6619a9601c08935b24d353f1360ca765795b178131517ecad4a71025216 Oct 30 23:41:54.943969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.943945 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" Oct 30 23:41:54.943969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.943950 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd" event={"ID":"2686bb47-640a-4dc2-81fb-945e17f98906","Type":"ContainerDied","Data":"f3d849704a3ee4369e6765a43a0b1584e5eccaf6ca9c44f1f4b69be0a35b49c3"} Oct 30 23:41:54.944172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.944004 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3d849704a3ee4369e6765a43a0b1584e5eccaf6ca9c44f1f4b69be0a35b49c3" Oct 30 23:41:54.945493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:54.945443 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" event={"ID":"fd65f927-691b-412a-9dcb-2b79fca2213e","Type":"ContainerStarted","Data":"838cf6619a9601c08935b24d353f1360ca765795b178131517ecad4a71025216"} Oct 30 23:41:55.725665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.725629 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798"] Oct 30 23:41:55.730369 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.730350 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.746021 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.745994 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798"] Oct 30 23:41:55.799228 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.799196 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbfwt\" (UniqueName: \"kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-kube-api-access-tbfwt\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.799421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.799256 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.799421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.799302 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.799421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.799341 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.799421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.799356 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.799421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.799384 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.799421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.799423 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.800028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.799578 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.800028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.799824 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.901352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.901301 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.901541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.901389 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.901541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.901415 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.901541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.901481 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.901541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.901532 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.901780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.901658 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.901780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.901728 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.901885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.901811 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.901885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.901847 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.902239 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.902210 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbfwt\" (UniqueName: \"kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-kube-api-access-tbfwt\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.902343 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.902283 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.902522 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.902215 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.902612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.902517 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.903167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.903146 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.905106 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.905085 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.905878 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.905855 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.924353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.924327 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbfwt\" (UniqueName: \"kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-kube-api-access-tbfwt\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.935440 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.935411 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:55.964949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.964885 7570 generic.go:356] "Generic (PLEG): container finished" podID="fd65f927-691b-412a-9dcb-2b79fca2213e" containerID="b56b05b193a948af227f890ca948b560cc2d06f9b102cf29c19182485a45487f" exitCode=0 Oct 30 23:41:55.964949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.964931 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" event={"ID":"fd65f927-691b-412a-9dcb-2b79fca2213e","Type":"ContainerDied","Data":"b56b05b193a948af227f890ca948b560cc2d06f9b102cf29c19182485a45487f"} Oct 30 23:41:55.973164 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.973123 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd"] Oct 30 23:41:55.979794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:55.979731 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.006767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.004729 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.006767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.004784 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6rc\" (UniqueName: \"kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-kube-api-access-xl6rc\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.006767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.004994 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.006767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.005042 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.006767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.005079 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74b8dd0-7067-44fd-94c0-948568ca285c-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.006767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.005148 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.006767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.005183 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.006767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.005208 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.006767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.005323 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.050809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.050773 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:41:56.170484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.166441 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.170484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.166519 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.170484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.166560 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74b8dd0-7067-44fd-94c0-948568ca285c-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.170484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.166634 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.170484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.166673 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.170484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.166702 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.170484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.166821 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.170484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.166932 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.170484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.166960 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6rc\" (UniqueName: \"kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-kube-api-access-xl6rc\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.170484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.167012 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.170484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.167391 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.170484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.167645 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.201515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.176699 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74b8dd0-7067-44fd-94c0-948568ca285c-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.201515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.177186 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.201515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.177234 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd"] Oct 30 23:41:56.255559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.255531 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2"] Oct 30 23:41:56.260258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.260195 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.261203 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.261155 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.261865 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.261845 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.263055 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.263038 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2"] Oct 30 23:41:56.264081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.263584 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.272046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.271997 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6rc\" (UniqueName: \"kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-kube-api-access-xl6rc\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.353930 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.345723 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:41:56.395895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.378032 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgpbv\" (UniqueName: \"kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-kube-api-access-lgpbv\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.395895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.378112 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.395895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.378191 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.395895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.378229 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.395895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.378271 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.395895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.378366 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.395895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.378389 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.395895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.378423 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.395895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.378604 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.442196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.442162 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6"] Oct 30 23:41:56.453281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.453259 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.473759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.473734 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6"] Oct 30 23:41:56.483341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.483309 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgpbv\" (UniqueName: \"kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-kube-api-access-lgpbv\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.483635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.483615 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.483764 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.483750 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.483856 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.483843 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q5t4\" (UniqueName: \"kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-kube-api-access-9q5t4\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.483985 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.483968 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.484163 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.484147 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.484275 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.484261 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.484388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.484369 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.484517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.484503 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.484738 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.484719 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.484847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.484832 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.484963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.484949 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.485160 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.485145 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.485345 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.485329 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.485456 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.485442 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.485655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.485638 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.485760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.485746 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.485870 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.485857 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.486740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.486717 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.487390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.487365 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.491399 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.491378 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.492177 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.491845 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.492311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.492061 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.648505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.616851 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.648505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.616905 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.648505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.616940 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.648505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.617034 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.648505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.617094 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q5t4\" (UniqueName: \"kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-kube-api-access-9q5t4\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.648505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.617136 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.648505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.617258 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.648505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.637932 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.648505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.641855 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.687524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.667062 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.687524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.667319 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.687524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.668751 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.687524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.668887 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.687524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.668994 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.687524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.669364 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.762490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.761637 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgpbv\" (UniqueName: \"kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-kube-api-access-lgpbv\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.762490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.762033 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.777842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.763696 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.777842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.764144 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.777842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.764905 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.777842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.765523 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q5t4\" (UniqueName: \"kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-kube-api-access-9q5t4\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.778596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.778306 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:41:56.845267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.820044 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:56.908785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:56.904719 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798"] Oct 30 23:41:57.088854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.088815 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:41:57.129955 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.106714 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:57.129955 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.125020 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:57.129955 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.125075 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:57.549706 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.549667 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd"] Oct 30 23:41:57.608354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.608315 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" event={"ID":"c9e446a0-c6da-40ab-aa1f-f3756e39619a","Type":"ContainerStarted","Data":"b22fff677a45cd41ef912f36f018f4e9da388b1fc09af78c9bd149702466e162"} Oct 30 23:41:57.743959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.743917 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:41:57.806542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.799246 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2"] Oct 30 23:41:57.902620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.835657 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:57.903003 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.900603 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:57.903177 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.900694 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:41:57.903604 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.903445 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:57.939484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:57.907838 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:58.002489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.001631 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:58.065586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.056597 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:58.264490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.260438 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:58.355488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.352283 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:58.359487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.355843 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:58.359877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.359839 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:41:58.383349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.383294 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:41:58.461828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.461781 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:58.462496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.462442 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:41:58.781828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.732183 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:41:58.781828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.746826 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:41:58.781828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.747934 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:58.781828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.747964 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:58.781828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.749294 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded" Oct 30 23:41:58.821331 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.821285 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:58.938095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.937966 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 30 23:41:58.953512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.951803 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:59.019354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:58.982780 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:41:59.019354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.010169 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:59.238665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.214055 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:41:59.240014 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.239990 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:41:59.634170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.390774 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:59.634170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.475769 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:41:59.634170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.476384 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:59.634170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.476414 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:41:59.634170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.551866 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:59.734485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.732595 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:59.872023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.766570 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:41:59.872023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.766993 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:59.872023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.813344 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:41:59.872023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.813763 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:41:59.872023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.813825 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:41:59.872023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.813863 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:41:59.872023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.813904 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:41:59.872023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.813942 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:41:59.872023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.813978 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:41:59.872023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.814014 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:41:59.872023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.814075 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:41:59.872023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.814108 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:41:59.897496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.884922 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:59.967488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.953573 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:41:59.967488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.958601 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:41:59.987849 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:41:59.987793 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:00.169426 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.169327 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" event={"ID":"a74b8dd0-7067-44fd-94c0-948568ca285c","Type":"ContainerStarted","Data":"a05def0e1f635f4aa81ef11eb9893b4ec5f7341b60b7943c9f780a81bf734901"} Oct 30 23:42:00.206876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.206819 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:00.229237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.207930 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:42:00.402163 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.397577 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" event={"ID":"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f","Type":"ContainerStarted","Data":"8d8209ac4e8f282dba722f237b62f035589e245b00c153fb5807e8ab1ec20e67"} Oct 30 23:42:00.427850 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:42:00.427755 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:42:00.483016 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.482537 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:00.504616 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.486677 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:00.504616 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.498643 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:00.504616 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.498750 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:00.616550 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.616500 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:01.024858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.626126 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:01.024858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.626173 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:01.024858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.626203 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:01.024858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.738788 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:01.024858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.767051 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:01.024858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.767196 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:42:01.024858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.911222 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:01.024858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.911898 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:01.024858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.954160 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:01.024858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:00.995010 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:01.260541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:01.236625 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:42:01.570740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:01.570704 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:42:01.717550 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:42:01.710760 7570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.111s" Oct 30 23:42:01.970149 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:01.965502 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:01.992489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:01.976600 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:01.992489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:01.977030 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:01.992489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:01.977315 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:01.992489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:01.977616 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:01.992489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:01.978623 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:42:01.992489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:01.978898 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:42:01.992489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:01.979171 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:42:01.992489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:01.979998 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:02.457553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.040644 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:02.590294 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.138581 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6"] Oct 30 23:42:02.668057 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.528588 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:42:02.692554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.532969 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:42:02.950159 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.950114 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:02.981098 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.981055 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:03.011412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.981416 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:03.012700 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.981556 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:03.013877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.981668 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:03.032583 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.981778 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:42:03.034058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.981886 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:03.034581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.982027 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:03.035741 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.982121 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:42:03.054735 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:02.982229 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:03.063541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.006638 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:03.074776 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.074678 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-xks-ca\") pod \"fd65f927-691b-412a-9dcb-2b79fca2213e\" (UID: \"fd65f927-691b-412a-9dcb-2b79fca2213e\") " Oct 30 23:42:03.187489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.180263 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-xks-ca" pod "fd65f927-691b-412a-9dcb-2b79fca2213e" (UID: "fd65f927-691b-412a-9dcb-2b79fca2213e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:03.264489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.246163 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd65f927-691b-412a-9dcb-2b79fca2213e-kube-api-access-4pwwf\" (UniqueName: \"kubernetes.io/projected/fd65f927-691b-412a-9dcb-2b79fca2213e-kube-api-access-4pwwf\") pod \"fd65f927-691b-412a-9dcb-2b79fca2213e\" (UID: \"fd65f927-691b-412a-9dcb-2b79fca2213e\") " Oct 30 23:42:03.265849 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.265810 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-openshift-ca\") pod \"fd65f927-691b-412a-9dcb-2b79fca2213e\" (UID: \"fd65f927-691b-412a-9dcb-2b79fca2213e\") " Oct 30 23:42:03.267786 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.267757 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:03.377661 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.342235 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-openshift-ca" pod "fd65f927-691b-412a-9dcb-2b79fca2213e" (UID: "fd65f927-691b-412a-9dcb-2b79fca2213e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:03.551978 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.551934 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd65f927-691b-412a-9dcb-2b79fca2213e-kube-api-access-4pwwf" pod "fd65f927-691b-412a-9dcb-2b79fca2213e" (UID: "fd65f927-691b-412a-9dcb-2b79fca2213e"). InnerVolumeSpecName "kube-api-access-4pwwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:03.844549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.807615 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:42:03.844549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.808039 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:42:03.844549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.808353 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:42:03.844549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.808794 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:03.844549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.808906 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:42:03.844549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.808935 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 23:42:03.864777 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.864723 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:03.939497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.922540 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pwwf\" (UniqueName: \"kubernetes.io/projected/fd65f927-691b-412a-9dcb-2b79fca2213e-kube-api-access-4pwwf\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:03.939877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:03.939847 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd65f927-691b-412a-9dcb-2b79fca2213e-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:04.062487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.041773 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:42:04.062487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.042372 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:04.062487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.042792 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:04.062487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.043129 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:04.062487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.043604 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:42:04.062487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.043928 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:04.062487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.044209 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:42:04.062487 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:42:04.056773 7570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.355s" Oct 30 23:42:04.062487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.056862 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:42:04.072490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.064580 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:04.099555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.076997 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:04.353420 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.353308 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" event={"ID":"c9e446a0-c6da-40ab-aa1f-f3756e39619a","Type":"ContainerStarted","Data":"2da79595c41ddcf813b52babb72238b7840eb347770a0979f860ea3c9781803d"} Oct 30 23:42:04.373490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.357508 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:42:04.466676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.466601 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:04.493588 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.375549 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" event={"ID":"fd65f927-691b-412a-9dcb-2b79fca2213e","Type":"ContainerDied","Data":"838cf6619a9601c08935b24d353f1360ca765795b178131517ecad4a71025216"} Oct 30 23:42:04.493881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.493856 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="838cf6619a9601c08935b24d353f1360ca765795b178131517ecad4a71025216" Oct 30 23:42:04.533541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.375529 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6" Oct 30 23:42:04.533541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.526167 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" event={"ID":"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984","Type":"ContainerStarted","Data":"c218cb1a1f5e8bf30f515d0d4dfbde6481fb1e84dfc8100da0b7f1098701f14c"} Oct 30 23:42:04.962557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.828582 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:04.962557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.828786 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:04.962557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.830346 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:04.962557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.890557 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:42:04.962557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:04.890614 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 23:42:05.007665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.007557 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:05.077487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.042968 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:05.078066 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.063481 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:42:05.078282 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.063602 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:42:05.078420 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.078395 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 23:42:05.078547 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.063671 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:42:05.078900 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.063642 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:05.079266 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.063705 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:42:05.079726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.063723 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:05.080089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.063741 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:42:05.080450 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.063759 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:05.152516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.128606 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:05.152516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.129973 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:42:05.152516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.132540 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:05.152516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.133346 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:42:05.152516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.133387 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 23:42:05.359850 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.359815 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97"] Oct 30 23:42:05.386485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.361279 7570 cpu_manager.go:414] "containerMap: removing container" podUID="fd65f927-691b-412a-9dcb-2b79fca2213e" containerName="revoke-keys" Oct 30 23:42:05.386759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.386732 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd65f927-691b-412a-9dcb-2b79fca2213e" containerName="revoke-keys" Oct 30 23:42:05.387293 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.387260 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd65f927-691b-412a-9dcb-2b79fca2213e" containerName="revoke-keys" Oct 30 23:42:05.389528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.379009 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:42:05.389730 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.389706 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 23:42:05.411518 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.401708 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:42:05.411518 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.401840 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:05.412841 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.412806 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:42:05.454079 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.454023 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97"] Oct 30 23:42:05.471488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.459344 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podStartSLOduration=9.459321203 podStartE2EDuration="9.459321203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:04.297956634 +0000 UTC m=+1335.816957727" watchObservedRunningTime="2026-10-30 23:42:05.459321203 +0000 UTC m=+1336.978322295" Oct 30 23:42:05.495488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.492627 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:05.517488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.510675 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:42:05.564488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.560606 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b727r\" (UniqueName: \"kubernetes.io/projected/1dbc604c-0936-4e1b-87c8-66d7e8397337-kube-api-access-b727r\") pod \"maas-api-revoke-sub-6192be83-fnp97\" (UID: \"1dbc604c-0936-4e1b-87c8-66d7e8397337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:05.572494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.565167 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6192be83-fnp97\" (UID: \"1dbc604c-0936-4e1b-87c8-66d7e8397337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:05.572874 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.572839 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6192be83-fnp97\" (UID: \"1dbc604c-0936-4e1b-87c8-66d7e8397337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:05.621967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.621864 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" event={"ID":"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984","Type":"ContainerStarted","Data":"7b62593c2ba5d2d1053164bdc55539dce34edcf66c3bf756b2febfddb756f124"} Oct 30 23:42:05.638493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.636110 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podStartSLOduration=10.636090278 podStartE2EDuration="10.636090278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:04.401060765 +0000 UTC m=+1335.920061867" watchObservedRunningTime="2026-10-30 23:42:05.636090278 +0000 UTC m=+1337.155091368" Oct 30 23:42:05.647553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.645025 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" event={"ID":"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f","Type":"ContainerStarted","Data":"c6a4e1f7e0b4586b2ac7fe2223526016c1728d8d667ebecad7ed2c062d5eae65"} Oct 30 23:42:05.690748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.676309 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" event={"ID":"a74b8dd0-7067-44fd-94c0-948568ca285c","Type":"ContainerStarted","Data":"ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5"} Oct 30 23:42:05.690748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.680859 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podStartSLOduration=10.680841777 podStartE2EDuration="10.680841777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:04.55236998 +0000 UTC m=+1336.071371073" watchObservedRunningTime="2026-10-30 23:42:05.680841777 +0000 UTC m=+1337.199842890" Oct 30 23:42:05.712348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.712302 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6192be83-fnp97\" (UID: \"1dbc604c-0936-4e1b-87c8-66d7e8397337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:05.737654 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.737606 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6192be83-fnp97\" (UID: \"1dbc604c-0936-4e1b-87c8-66d7e8397337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:05.738227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.738194 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b727r\" (UniqueName: \"kubernetes.io/projected/1dbc604c-0936-4e1b-87c8-66d7e8397337-kube-api-access-b727r\") pod \"maas-api-revoke-sub-6192be83-fnp97\" (UID: \"1dbc604c-0936-4e1b-87c8-66d7e8397337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:05.771737 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.714268 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6192be83-fnp97\" (UID: \"1dbc604c-0936-4e1b-87c8-66d7e8397337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:05.772048 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.742285 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podStartSLOduration=9.742267463 podStartE2EDuration="9.742267463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:04.606323828 +0000 UTC m=+1336.125324921" watchObservedRunningTime="2026-10-30 23:42:05.742267463 +0000 UTC m=+1337.261268554" Oct 30 23:42:05.797488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.781493 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:42:05.797488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.781677 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 23:42:05.824334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.824299 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b727r\" (UniqueName: \"kubernetes.io/projected/1dbc604c-0936-4e1b-87c8-66d7e8397337-kube-api-access-b727r\") pod \"maas-api-revoke-sub-6192be83-fnp97\" (UID: \"1dbc604c-0936-4e1b-87c8-66d7e8397337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:05.880987 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.826955 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6192be83-fnp97\" (UID: \"1dbc604c-0936-4e1b-87c8-66d7e8397337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:05.880987 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.835270 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:42:05.880987 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.835350 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:05.880987 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:05.835373 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:42:06.093497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.076957 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:06.093497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.077346 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:42:06.093497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.077389 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 23:42:06.093497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.078079 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:06.093497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.078840 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:06.153968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.153865 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:06.201796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.201742 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:06.232490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.207976 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:06.297060 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.270546 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:06.312507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.270584 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:06.315987 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.270594 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:42:06.316982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.270608 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:42:06.326239 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.270618 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:06.428077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.427980 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:42:06.428356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.428307 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 23:42:06.488939 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.488812 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:42:06.489356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.489211 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 23:42:06.797393 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.797275 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:42:06.797807 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:06.797772 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 23:42:07.073680 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.073573 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:42:07.109868 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.109833 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:42:07.110487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.110436 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 23:42:07.160735 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.156513 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:07.212354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.183340 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:42:07.243094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.213261 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:07.267486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.213947 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:07.372493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.237972 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:07.377619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.309937 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:42:07.384488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.309980 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:07.384488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.310183 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:07.384488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.325843 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:07.384488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.326057 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:07.384488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.347167 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:07.510491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.368678 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:42:07.510491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.505714 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 23:42:07.580537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.580500 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:42:07.580835 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.580793 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 23:42:07.759555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.751073 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 30 23:42:07.812538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.810726 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:42:07.812538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.810792 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 23:42:07.868046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:07.867991 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:08.077046 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:42:07.879497 7570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.187s" Oct 30 23:42:08.107704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.107660 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:08.144490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.124840 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:08.318983 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.168957 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:42:08.319287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.319251 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 23:42:08.319436 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.284669 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:42:08.319708 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.284582 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:08.319928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.284699 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:42:08.353841 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:42:08.353803 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbc604c_0936_4e1b_87c8_66d7e8397337.slice/crio-9a57b22bfa83380afe5c8098509be9342ae63f566dedd3b5c212d75d8a636eb9 WatchSource:0}: Error finding container 9a57b22bfa83380afe5c8098509be9342ae63f566dedd3b5c212d75d8a636eb9: Status 404 returned error can't find the container with id 9a57b22bfa83380afe5c8098509be9342ae63f566dedd3b5c212d75d8a636eb9 Oct 30 23:42:08.416521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.412178 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:08.416521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.413560 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:08.416521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.413714 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:42:08.416521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.413810 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:08.416521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.413905 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:08.416521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.413988 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:42:08.416521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.414087 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:42:08.416521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.414322 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:08.663928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.663832 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:42:08.669489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.665987 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 23:42:08.669760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.664178 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:08.715400 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.715362 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:42:08.716273 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.716238 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 23:42:08.793656 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.793612 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:42:08.793966 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.793922 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 23:42:08.978160 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:08.978107 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.088504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.077624 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:42:09.088504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.077694 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 23:42:09.088504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.086717 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" event={"ID":"1dbc604c-0936-4e1b-87c8-66d7e8397337","Type":"ContainerStarted","Data":"9a57b22bfa83380afe5c8098509be9342ae63f566dedd3b5c212d75d8a636eb9"} Oct 30 23:42:09.088504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.086787 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97"] Oct 30 23:42:09.115546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.109613 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:42:09.115546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.109681 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 23:42:09.125599 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.125554 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.321714 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.321582 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.338496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.325926 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.370515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.356592 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.409722 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.409685 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:42:09.490622 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.490573 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 23:42:09.494576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.472068 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.519922 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.472115 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.520384 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.472151 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.520799 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.472167 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.521243 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.472188 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.695426 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.472213 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.695953 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.472232 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.696328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.431679 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.696686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.560792 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:09.827179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.827142 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:42:09.827626 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.827595 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 23:42:09.875304 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:09.875266 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:42:10.117892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.117862 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:42:10.118192 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.118147 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 23:42:10.155490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.129921 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:10.170518 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.156359 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:42:10.244228 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.244180 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 23:42:10.244512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.212643 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:10.350552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.325745 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:10.375594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.365352 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:42:10.375594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.365408 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 23:42:10.375594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.365499 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:10.414487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.382079 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:10.515503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.514575 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:10.530486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.519696 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:10.555785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.546585 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:10.555785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.546960 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:10.564503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.556415 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:10.564859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.564825 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:10.697198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.697098 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:42:10.734489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.702641 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:10.734489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.703031 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:10.782845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.782811 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:42:10.783137 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:10.783107 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 23:42:11.089574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.085546 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp"] Oct 30 23:42:11.245025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.162607 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:42:11.245025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.162676 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 23:42:11.245025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.164163 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:42:11.245025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.183548 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:42:11.245025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.195045 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:42:11.245025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.195098 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 23:42:11.304404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.269242 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp"] Oct 30 23:42:11.304404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.269301 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:42:11.304404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.270380 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.359548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.354975 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:11.367699 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.367669 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:42:11.370451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.370425 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:42:11.370708 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.370673 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 23:42:11.370907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.370885 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:11.386095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.386059 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:11.450499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.449056 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.450499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.449144 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.450499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.449179 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4zm9\" (UniqueName: \"kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-kube-api-access-m4zm9\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.450499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.449724 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.459486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.457566 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c25f85-fbde-4c00-94c9-502637e57ba5-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.459486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.457624 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.459486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.457659 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.459486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.457754 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.459486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.457911 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.493499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.493124 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" event={"ID":"1dbc604c-0936-4e1b-87c8-66d7e8397337","Type":"ContainerStarted","Data":"9c13400c2777f77ae5d445ac2913dff5775d73a94f55442fa3e1968275bae38c"} Oct 30 23:42:11.521551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.521412 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:11.560791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.560750 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.563827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.563801 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.564341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.564322 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.565433 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.565400 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4zm9\" (UniqueName: \"kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-kube-api-access-m4zm9\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.578878 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.565106 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.579047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.561696 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:11.580206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.561664 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:11.580206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.561708 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:11.580206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.579087 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.580206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.580079 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c25f85-fbde-4c00-94c9-502637e57ba5-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.580206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.580128 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.580206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.580164 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.580619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.580261 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.583492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.581354 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c25f85-fbde-4c00-94c9-502637e57ba5-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.611093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.611013 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.611680 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.611657 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.614984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.614932 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.624679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.615537 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.625445 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.625421 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.641521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.635418 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" podStartSLOduration=8.635383856 podStartE2EDuration="8.635383856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:11.539584709 +0000 UTC m=+1343.058585802" watchObservedRunningTime="2026-10-30 23:42:11.635383856 +0000 UTC m=+1343.154384947" Oct 30 23:42:11.658494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.658050 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.714406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.713363 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:11.736062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.714677 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:42:11.736062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.734000 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:42:11.768490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.766160 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4zm9\" (UniqueName: \"kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-kube-api-access-m4zm9\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:11.785146 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.785104 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:42:11.806593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.786406 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:42:11.806999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.806876 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 23:42:11.808826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:11.808803 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:12.053782 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.053294 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:42:12.053782 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.053359 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 23:42:12.107608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.107569 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:42:12.135341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.134845 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:42:12.135341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.134901 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 23:42:12.214526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.204395 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:42:12.214526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.205050 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:12.214526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.205309 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:42:12.367170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.365804 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:12.383659 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.382827 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:42:12.383659 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.382884 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 23:42:12.404564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.397937 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:42:12.445801 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.429614 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:42:12.448848 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.448410 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:12.523495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.522880 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:12.535347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.535312 7570 generic.go:356] "Generic (PLEG): container finished" podID="1dbc604c-0936-4e1b-87c8-66d7e8397337" containerID="9c13400c2777f77ae5d445ac2913dff5775d73a94f55442fa3e1968275bae38c" exitCode=0 Oct 30 23:42:12.535613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.535592 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" event={"ID":"1dbc604c-0936-4e1b-87c8-66d7e8397337","Type":"ContainerDied","Data":"9c13400c2777f77ae5d445ac2913dff5775d73a94f55442fa3e1968275bae38c"} Oct 30 23:42:12.606602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.606533 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:12.652510 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.649995 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:12.659560 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.655770 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:12.723610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.723110 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:12.736845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.734555 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:42:12.743378 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.743344 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:12.819488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.814677 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp"] Oct 30 23:42:12.840488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.823856 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:42:12.840488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.823926 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 23:42:12.898773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.893981 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:42:12.926748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.913606 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:12.926748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.913758 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:12.926748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:12.913850 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:13.383645 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.383606 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:42:13.387577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.387536 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 23:42:13.508488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.504228 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:42:13.523718 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.523621 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:13.532286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.532258 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:42:13.598057 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.598010 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:42:13.598711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.598681 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:42:13.599171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.599147 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 23:42:13.599946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.599919 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:13.655742 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.654140 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:42:13.727194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.727160 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:42:13.765489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.740017 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:42:13.964998 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.964967 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:42:13.965273 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.965254 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:42:13.965423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.965410 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:42:13.969023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.965561 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:42:13.969023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.799860 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:42:13.969023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.966146 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 23:42:13.969023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:13.822694 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:14.003819 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.002652 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:42:14.085704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.034932 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:42:14.085704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.065855 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:42:14.101926 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.086273 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:42:14.101926 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.086402 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:42:14.114194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.114166 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" event={"ID":"61c25f85-fbde-4c00-94c9-502637e57ba5","Type":"ContainerStarted","Data":"61ddd0abdafe7f5014952ede2949d96149fe2fbd16e4bad29537b87f0d8b2a71"} Oct 30 23:42:14.379593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.379059 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:42:14.522551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.522518 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:42:14.554766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.547611 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:42:14.683746 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.683667 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:14.745498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.745421 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-openshift-ca\") pod \"1dbc604c-0936-4e1b-87c8-66d7e8397337\" (UID: \"1dbc604c-0936-4e1b-87c8-66d7e8397337\") " Oct 30 23:42:14.748274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.745870 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-xks-ca\") pod \"1dbc604c-0936-4e1b-87c8-66d7e8397337\" (UID: \"1dbc604c-0936-4e1b-87c8-66d7e8397337\") " Oct 30 23:42:14.748274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.745996 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbc604c-0936-4e1b-87c8-66d7e8397337-kube-api-access-b727r\" (UniqueName: \"kubernetes.io/projected/1dbc604c-0936-4e1b-87c8-66d7e8397337-kube-api-access-b727r\") pod \"1dbc604c-0936-4e1b-87c8-66d7e8397337\" (UID: \"1dbc604c-0936-4e1b-87c8-66d7e8397337\") " Oct 30 23:42:14.748274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.748060 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-openshift-ca" pod "1dbc604c-0936-4e1b-87c8-66d7e8397337" (UID: "1dbc604c-0936-4e1b-87c8-66d7e8397337"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:14.751502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.751477 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbc604c-0936-4e1b-87c8-66d7e8397337-kube-api-access-b727r" pod "1dbc604c-0936-4e1b-87c8-66d7e8397337" (UID: "1dbc604c-0936-4e1b-87c8-66d7e8397337"). InnerVolumeSpecName "kube-api-access-b727r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:14.752206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.752179 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-xks-ca" pod "1dbc604c-0936-4e1b-87c8-66d7e8397337" (UID: "1dbc604c-0936-4e1b-87c8-66d7e8397337"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:14.786180 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.786147 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:42:14.788694 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.788670 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:42:14.849236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.849198 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:14.849236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.849232 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b727r\" (UniqueName: \"kubernetes.io/projected/1dbc604c-0936-4e1b-87c8-66d7e8397337-kube-api-access-b727r\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:14.849435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:14.849248 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbc604c-0936-4e1b-87c8-66d7e8397337-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:15.147435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.147398 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr"] Oct 30 23:42:15.150039 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.149813 7570 cpu_manager.go:414] "containerMap: removing container" podUID="1dbc604c-0936-4e1b-87c8-66d7e8397337" containerName="revoke-keys" Oct 30 23:42:15.150234 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.150212 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbc604c-0936-4e1b-87c8-66d7e8397337" containerName="revoke-keys" Oct 30 23:42:15.150785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.150750 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbc604c-0936-4e1b-87c8-66d7e8397337" containerName="revoke-keys" Oct 30 23:42:15.166199 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.166171 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:15.176197 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.176008 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr"] Oct 30 23:42:15.182942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.181453 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" event={"ID":"61c25f85-fbde-4c00-94c9-502637e57ba5","Type":"ContainerStarted","Data":"9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b"} Oct 30 23:42:15.185758 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.185734 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" Oct 30 23:42:15.185974 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.185951 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97" event={"ID":"1dbc604c-0936-4e1b-87c8-66d7e8397337","Type":"ContainerDied","Data":"9a57b22bfa83380afe5c8098509be9342ae63f566dedd3b5c212d75d8a636eb9"} Oct 30 23:42:15.186078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.185995 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a57b22bfa83380afe5c8098509be9342ae63f566dedd3b5c212d75d8a636eb9" Oct 30 23:42:15.254854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.254786 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podStartSLOduration=5.254771183 podStartE2EDuration="5.254771183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:15.250100636 +0000 UTC m=+1346.769101727" watchObservedRunningTime="2026-10-30 23:42:15.254771183 +0000 UTC m=+1346.773772325" Oct 30 23:42:15.275501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.275446 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c82df7b-cnbsr\" (UID: \"fefad00c-f5e3-490c-a477-8a7f643ba668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:15.275684 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.275513 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c82df7b-cnbsr\" (UID: \"fefad00c-f5e3-490c-a477-8a7f643ba668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:15.276166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.276134 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjrt\" (UniqueName: \"kubernetes.io/projected/fefad00c-f5e3-490c-a477-8a7f643ba668-kube-api-access-swjrt\") pod \"maas-api-revoke-sub-5c82df7b-cnbsr\" (UID: \"fefad00c-f5e3-490c-a477-8a7f643ba668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:15.378011 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.377968 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjrt\" (UniqueName: \"kubernetes.io/projected/fefad00c-f5e3-490c-a477-8a7f643ba668-kube-api-access-swjrt\") pod \"maas-api-revoke-sub-5c82df7b-cnbsr\" (UID: \"fefad00c-f5e3-490c-a477-8a7f643ba668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:15.378198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.378150 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c82df7b-cnbsr\" (UID: \"fefad00c-f5e3-490c-a477-8a7f643ba668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:15.378198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.378172 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c82df7b-cnbsr\" (UID: \"fefad00c-f5e3-490c-a477-8a7f643ba668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:15.378913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.378880 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c82df7b-cnbsr\" (UID: \"fefad00c-f5e3-490c-a477-8a7f643ba668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:15.380862 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.380840 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c82df7b-cnbsr\" (UID: \"fefad00c-f5e3-490c-a477-8a7f643ba668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:15.411425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.411325 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjrt\" (UniqueName: \"kubernetes.io/projected/fefad00c-f5e3-490c-a477-8a7f643ba668-kube-api-access-swjrt\") pod \"maas-api-revoke-sub-5c82df7b-cnbsr\" (UID: \"fefad00c-f5e3-490c-a477-8a7f643ba668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:15.539388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.539348 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:15.817947 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.817847 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:15.819089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.819062 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:42:15.819224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.819119 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 23:42:15.857708 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:42:15.857678 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefad00c_f5e3_490c_a477_8a7f643ba668.slice/crio-2c05175105cfebf0ff75d598143bf176bb44b8d70bdfbd3d51b956b998723745 WatchSource:0}: Error finding container 2c05175105cfebf0ff75d598143bf176bb44b8d70bdfbd3d51b956b998723745: Status 404 returned error can't find the container with id 2c05175105cfebf0ff75d598143bf176bb44b8d70bdfbd3d51b956b998723745 Oct 30 23:42:15.860143 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:15.860114 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr"] Oct 30 23:42:16.219832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:16.219796 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" event={"ID":"fefad00c-f5e3-490c-a477-8a7f643ba668","Type":"ContainerStarted","Data":"2c05175105cfebf0ff75d598143bf176bb44b8d70bdfbd3d51b956b998723745"} Oct 30 23:42:16.832570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:16.823872 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:42:16.832570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:16.823933 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 23:42:17.428563 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:17.423519 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" event={"ID":"fefad00c-f5e3-490c-a477-8a7f643ba668","Type":"ContainerStarted","Data":"21d7b02cbb0da5227ca3463e2496a43a766a74b2b444c25926e011b95f38766d"} Oct 30 23:42:17.828553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:17.825859 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:42:17.828553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:17.825921 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 23:42:18.621534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:18.575563 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded" Oct 30 23:42:18.621534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:18.620356 7570 generic.go:356] "Generic (PLEG): container finished" podID="fefad00c-f5e3-490c-a477-8a7f643ba668" containerID="21d7b02cbb0da5227ca3463e2496a43a766a74b2b444c25926e011b95f38766d" exitCode=0 Oct 30 23:42:18.621534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:18.620500 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" event={"ID":"fefad00c-f5e3-490c-a477-8a7f643ba668","Type":"ContainerDied","Data":"21d7b02cbb0da5227ca3463e2496a43a766a74b2b444c25926e011b95f38766d"} Oct 30 23:42:18.698489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:18.697604 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:42:18.763585 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:18.755876 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:18.832559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:18.820931 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:42:18.832559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:18.820999 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 23:42:18.855611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:18.855561 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 30 23:42:18.918967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:18.918862 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:19.001357 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.001308 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:19.014485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.001714 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:19.050502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.001962 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:19.080518 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.001992 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:19.080518 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.002013 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:19.080518 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.002039 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:19.080518 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.025547 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:42:19.080518 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.078780 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:19.234547 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.234305 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:19.505761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.505571 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Oct 30 23:42:19.956681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.818548 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:42:19.956681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.819205 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:42:19.956681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.821093 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:19.956681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.821568 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:42:19.956681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.829817 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:42:19.956681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.829868 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 23:42:19.956681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.941261 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:19.956681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:19.942871 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:42:20.101879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.064781 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:20.101879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.065407 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:42:20.101879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.070565 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:20.101879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.072065 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:42:20.101879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.073161 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:20.101879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.073550 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:42:20.101879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.074278 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:20.101879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.075043 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:20.101879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.075780 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:42:20.101879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.075835 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:42:20.101879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.076423 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:42:20.101879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.093618 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:42:20.219645 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.110803 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:20.219645 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.111235 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:42:20.228508 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.221521 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:20.228508 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.221955 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:42:20.251505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.229408 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:20.251505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.242498 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:42:20.293589 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.265237 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:42:20.562557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.488974 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:20.837818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.837174 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:20.837818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.837385 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:42:20.965698 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.906916 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:42:20.965698 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.906973 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 23:42:20.965698 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:20.951506 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:21.212652 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.065553 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:21.212652 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.094583 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:21.212652 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.094917 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:21.212652 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.111760 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:21.298179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.298074 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:42:21.355643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.348219 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:21.355643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.348541 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:21.355643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.348762 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:42:21.597356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.597320 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:42:21.853013 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.852909 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:21.853623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.853592 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:21.941491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.937693 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:42:21.973521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:21.968488 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:22.074492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.030141 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:22.113012 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.112903 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:22.113948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.113915 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:22.134824 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.134770 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-openshift-ca\") pod \"fefad00c-f5e3-490c-a477-8a7f643ba668\" (UID: \"fefad00c-f5e3-490c-a477-8a7f643ba668\") " Oct 30 23:42:22.154129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.154099 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefad00c-f5e3-490c-a477-8a7f643ba668-kube-api-access-swjrt\" (UniqueName: \"kubernetes.io/projected/fefad00c-f5e3-490c-a477-8a7f643ba668-kube-api-access-swjrt\") pod \"fefad00c-f5e3-490c-a477-8a7f643ba668\" (UID: \"fefad00c-f5e3-490c-a477-8a7f643ba668\") " Oct 30 23:42:22.173053 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.173011 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-xks-ca\") pod \"fefad00c-f5e3-490c-a477-8a7f643ba668\" (UID: \"fefad00c-f5e3-490c-a477-8a7f643ba668\") " Oct 30 23:42:22.174474 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.157102 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:22.184748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.157133 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:42:22.186951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.155637 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-openshift-ca" pod "fefad00c-f5e3-490c-a477-8a7f643ba668" (UID: "fefad00c-f5e3-490c-a477-8a7f643ba668"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:22.187395 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.187348 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" event={"ID":"fefad00c-f5e3-490c-a477-8a7f643ba668","Type":"ContainerDied","Data":"2c05175105cfebf0ff75d598143bf176bb44b8d70bdfbd3d51b956b998723745"} Oct 30 23:42:22.187577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.187559 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c05175105cfebf0ff75d598143bf176bb44b8d70bdfbd3d51b956b998723745" Oct 30 23:42:22.238759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.238617 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:42:22.239711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.239037 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 23:42:22.538515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.500196 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:42:22.560487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.554821 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefad00c-f5e3-490c-a477-8a7f643ba668-kube-api-access-swjrt" pod "fefad00c-f5e3-490c-a477-8a7f643ba668" (UID: "fefad00c-f5e3-490c-a477-8a7f643ba668"). InnerVolumeSpecName "kube-api-access-swjrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:22.593891 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.593841 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swjrt\" (UniqueName: \"kubernetes.io/projected/fefad00c-f5e3-490c-a477-8a7f643ba668-kube-api-access-swjrt\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:22.681210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.681101 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:22.747234 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.632197 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:42:22.765906 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.632597 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:22.766805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.745796 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:22.767003 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.631974 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:22.972761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.890096 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:22.972761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.947719 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:22.984636 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:22.984585 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-xks-ca" pod "fefad00c-f5e3-490c-a477-8a7f643ba668" (UID: "fefad00c-f5e3-490c-a477-8a7f643ba668"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:23.047543 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.036308 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:42:23.057337 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.057295 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 23:42:23.065298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.064566 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:23.065664 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.065523 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:23.083390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.083363 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefad00c-f5e3-490c-a477-8a7f643ba668-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:23.096982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.096955 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:42:23.097231 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.097217 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:42:23.097399 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.097387 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:42:23.097551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.097539 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:42:23.097700 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.097688 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:42:23.097893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.097881 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:42:23.106864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.106308 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:42:23.116589 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.116559 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:23.137079 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.115649 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:23.177422 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.177380 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:23.243601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.243427 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:42:23.246821 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.246792 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:42:23.254387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.254360 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr" Oct 30 23:42:23.255554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.255532 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:42:23.262980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.262953 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:42:23.264933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.264908 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:42:23.265971 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.265953 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:42:23.822619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.821182 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:42:23.822619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:23.821257 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 23:42:24.242603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.242568 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2"] Oct 30 23:42:24.243317 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.243300 7570 cpu_manager.go:414] "containerMap: removing container" podUID="fefad00c-f5e3-490c-a477-8a7f643ba668" containerName="revoke-keys" Oct 30 23:42:24.243317 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.243316 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefad00c-f5e3-490c-a477-8a7f643ba668" containerName="revoke-keys" Oct 30 23:42:24.243502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.243411 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefad00c-f5e3-490c-a477-8a7f643ba668" containerName="revoke-keys" Oct 30 23:42:24.247729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.247708 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:24.257280 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.257258 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:42:24.257732 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.257705 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:42:24.272245 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.272219 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2"] Oct 30 23:42:24.370273 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.370234 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5106cbe-bj7p2\" (UID: \"85e518bd-b5ec-4504-85fa-8c012807bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:24.370449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.370389 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5zl\" (UniqueName: \"kubernetes.io/projected/85e518bd-b5ec-4504-85fa-8c012807bbe0-kube-api-access-xs5zl\") pod \"maas-api-revoke-sub-a5106cbe-bj7p2\" (UID: \"85e518bd-b5ec-4504-85fa-8c012807bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:24.370449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.370443 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5106cbe-bj7p2\" (UID: \"85e518bd-b5ec-4504-85fa-8c012807bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:24.471770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.471732 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5106cbe-bj7p2\" (UID: \"85e518bd-b5ec-4504-85fa-8c012807bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:24.471969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.471869 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5zl\" (UniqueName: \"kubernetes.io/projected/85e518bd-b5ec-4504-85fa-8c012807bbe0-kube-api-access-xs5zl\") pod \"maas-api-revoke-sub-a5106cbe-bj7p2\" (UID: \"85e518bd-b5ec-4504-85fa-8c012807bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:24.471969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.471938 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5106cbe-bj7p2\" (UID: \"85e518bd-b5ec-4504-85fa-8c012807bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:24.472764 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.472729 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5106cbe-bj7p2\" (UID: \"85e518bd-b5ec-4504-85fa-8c012807bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:24.477483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.474646 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5106cbe-bj7p2\" (UID: \"85e518bd-b5ec-4504-85fa-8c012807bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:24.498096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.498024 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5zl\" (UniqueName: \"kubernetes.io/projected/85e518bd-b5ec-4504-85fa-8c012807bbe0-kube-api-access-xs5zl\") pod \"maas-api-revoke-sub-a5106cbe-bj7p2\" (UID: \"85e518bd-b5ec-4504-85fa-8c012807bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:24.565588 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.565556 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:24.819257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.819178 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:42:24.819257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.819231 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 23:42:24.907597 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:42:24.901378 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e518bd_b5ec_4504_85fa_8c012807bbe0.slice/crio-3c98ff09ee8ceef974f62834f15e348dba26ab5e58f50fc056e3a23cea27154b WatchSource:0}: Error finding container 3c98ff09ee8ceef974f62834f15e348dba26ab5e58f50fc056e3a23cea27154b: Status 404 returned error can't find the container with id 3c98ff09ee8ceef974f62834f15e348dba26ab5e58f50fc056e3a23cea27154b Oct 30 23:42:24.907597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:24.904889 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2"] Oct 30 23:42:25.284001 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:25.283965 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" event={"ID":"85e518bd-b5ec-4504-85fa-8c012807bbe0","Type":"ContainerStarted","Data":"3c98ff09ee8ceef974f62834f15e348dba26ab5e58f50fc056e3a23cea27154b"} Oct 30 23:42:26.115006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.114975 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs"] Oct 30 23:42:26.119711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.119689 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.175331 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.175287 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs"] Oct 30 23:42:26.182287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.182257 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx"] Oct 30 23:42:26.188327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.188302 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.222408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.222366 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.229626 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.229593 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kgr2\" (UniqueName: \"kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-kube-api-access-7kgr2\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.229804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.229653 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.229804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.229769 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.229934 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.229846 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.229995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.229946 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e618c702-d336-43d9-8857-430b4fa5b8ce-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.230227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.230187 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.230369 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.230348 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.230617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.230447 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.230617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.230530 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx"] Oct 30 23:42:26.295498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.295402 7570 generic.go:356] "Generic (PLEG): container finished" podID="85e518bd-b5ec-4504-85fa-8c012807bbe0" containerID="f5ba846fca363916029b861af4486c90fdc98342c7fd3c4872d8d350d0650c0c" exitCode=0 Oct 30 23:42:26.295892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.295548 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" event={"ID":"85e518bd-b5ec-4504-85fa-8c012807bbe0","Type":"ContainerDied","Data":"f5ba846fca363916029b861af4486c90fdc98342c7fd3c4872d8d350d0650c0c"} Oct 30 23:42:26.333257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.332717 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e618c702-d336-43d9-8857-430b4fa5b8ce-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.333257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.332782 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.333257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.332844 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570b6b11-715d-4a67-9360-78aa45580d57-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.333257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.332930 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.333257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333022 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.333257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333055 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.333257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333097 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.333257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333191 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6td7j\" (UniqueName: \"kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-kube-api-access-6td7j\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.333257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333240 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.334009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333301 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570b6b11-715d-4a67-9360-78aa45580d57-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.334009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333570 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.334009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333624 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.334009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333662 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.334009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333699 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.334009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333740 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kgr2\" (UniqueName: \"kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-kube-api-access-7kgr2\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.334009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333776 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.334009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333784 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.334009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333809 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.334009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.333890 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.334617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.334026 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.334617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.334191 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e618c702-d336-43d9-8857-430b4fa5b8ce-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.334617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.334447 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.335039 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.335014 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.339819 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.339790 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.340162 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.340138 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r"] Oct 30 23:42:26.346396 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.346377 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:26.348552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.348533 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.420531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.419500 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r"] Oct 30 23:42:26.429719 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.429687 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.434920 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.432686 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kgr2\" (UniqueName: \"kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-kube-api-access-7kgr2\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.435661 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.435616 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a76e6f62-rdp9r\" (UID: \"520a1cb3-77dc-47d6-add9-0c262b514835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:26.435781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.435688 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570b6b11-715d-4a67-9360-78aa45580d57-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.435781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.435762 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.435900 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.435820 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgs5x\" (UniqueName: \"kubernetes.io/projected/520a1cb3-77dc-47d6-add9-0c262b514835-kube-api-access-hgs5x\") pod \"maas-api-revoke-sub-a76e6f62-rdp9r\" (UID: \"520a1cb3-77dc-47d6-add9-0c262b514835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:26.435900 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.435891 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.436013 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.435926 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a76e6f62-rdp9r\" (UID: \"520a1cb3-77dc-47d6-add9-0c262b514835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:26.436208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.436168 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.436285 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.436250 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570b6b11-715d-4a67-9360-78aa45580d57-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.436397 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.436379 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.436586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.436566 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.436665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.436617 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.436735 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.436715 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6td7j\" (UniqueName: \"kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-kube-api-access-6td7j\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.437951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.437926 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570b6b11-715d-4a67-9360-78aa45580d57-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.438242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.438219 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.438981 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.438957 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.439336 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.439314 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.444946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.444878 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570b6b11-715d-4a67-9360-78aa45580d57-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.446505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.446328 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:26.480355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.479995 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.505053 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.505021 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.508637 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.507360 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6td7j\" (UniqueName: \"kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-kube-api-access-6td7j\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.512653 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.512620 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.533506 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.533203 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:26.538800 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.538755 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a76e6f62-rdp9r\" (UID: \"520a1cb3-77dc-47d6-add9-0c262b514835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:26.539838 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.539687 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a76e6f62-rdp9r\" (UID: \"520a1cb3-77dc-47d6-add9-0c262b514835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:26.540314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.540292 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgs5x\" (UniqueName: \"kubernetes.io/projected/520a1cb3-77dc-47d6-add9-0c262b514835-kube-api-access-hgs5x\") pod \"maas-api-revoke-sub-a76e6f62-rdp9r\" (UID: \"520a1cb3-77dc-47d6-add9-0c262b514835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:26.540540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.540519 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a76e6f62-rdp9r\" (UID: \"520a1cb3-77dc-47d6-add9-0c262b514835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:26.550129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.550053 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a76e6f62-rdp9r\" (UID: \"520a1cb3-77dc-47d6-add9-0c262b514835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:26.560823 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.560765 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgs5x\" (UniqueName: \"kubernetes.io/projected/520a1cb3-77dc-47d6-add9-0c262b514835-kube-api-access-hgs5x\") pod \"maas-api-revoke-sub-a76e6f62-rdp9r\" (UID: \"520a1cb3-77dc-47d6-add9-0c262b514835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:26.747059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.746827 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:26.817956 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.817848 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:26.828714 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.828062 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:42:26.828714 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.828120 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:26.869750 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.869256 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs"] Oct 30 23:42:26.870294 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:42:26.870233 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode618c702_d336_43d9_8857_430b4fa5b8ce.slice/crio-bb833b9595500cffccb95ab2ef0970cf8cc9c5c250b6057db1fd00db7d665d61 WatchSource:0}: Error finding container bb833b9595500cffccb95ab2ef0970cf8cc9c5c250b6057db1fd00db7d665d61: Status 404 returned error can't find the container with id bb833b9595500cffccb95ab2ef0970cf8cc9c5c250b6057db1fd00db7d665d61 Oct 30 23:42:26.876684 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.873330 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:26.876684 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.873417 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:26.876684 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.873508 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:26.923647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.923615 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:26.930771 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.929920 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx"] Oct 30 23:42:26.934140 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:42:26.934111 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570b6b11_715d_4a67_9360_78aa45580d57.slice/crio-41197724134ea2933938575acc176957120bf0d1f27a419c0c7a6e8711116922 WatchSource:0}: Error finding container 41197724134ea2933938575acc176957120bf0d1f27a419c0c7a6e8711116922: Status 404 returned error can't find the container with id 41197724134ea2933938575acc176957120bf0d1f27a419c0c7a6e8711116922 Oct 30 23:42:26.937036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.937004 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:26.937127 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.937090 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:26.937127 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:26.937120 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:27.094201 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.094163 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r"] Oct 30 23:42:27.320823 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.320771 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" event={"ID":"e618c702-d336-43d9-8857-430b4fa5b8ce","Type":"ContainerStarted","Data":"e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf"} Oct 30 23:42:27.321360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.320890 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" event={"ID":"e618c702-d336-43d9-8857-430b4fa5b8ce","Type":"ContainerStarted","Data":"bb833b9595500cffccb95ab2ef0970cf8cc9c5c250b6057db1fd00db7d665d61"} Oct 30 23:42:27.323448 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.323417 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" event={"ID":"520a1cb3-77dc-47d6-add9-0c262b514835","Type":"ContainerStarted","Data":"0935314a78121e9c73ab62205f0feb566847c14b2db56bd8c96fd92b25dda434"} Oct 30 23:42:27.325756 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.325726 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" event={"ID":"570b6b11-715d-4a67-9360-78aa45580d57","Type":"ContainerStarted","Data":"f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626"} Oct 30 23:42:27.325859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.325782 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" event={"ID":"570b6b11-715d-4a67-9360-78aa45580d57","Type":"ContainerStarted","Data":"41197724134ea2933938575acc176957120bf0d1f27a419c0c7a6e8711116922"} Oct 30 23:42:27.327509 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.327442 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:42:27.362552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.358956 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" podStartSLOduration=1.358937069 podStartE2EDuration="1.358937069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:27.355757729 +0000 UTC m=+1358.874758850" watchObservedRunningTime="2026-10-30 23:42:27.358937069 +0000 UTC m=+1358.877938160" Oct 30 23:42:27.437938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.437581 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podStartSLOduration=1.437559673 podStartE2EDuration="1.437559673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:27.43481127 +0000 UTC m=+1358.953812360" watchObservedRunningTime="2026-10-30 23:42:27.437559673 +0000 UTC m=+1358.956560767" Oct 30 23:42:27.447005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.446971 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:27.448702 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.448673 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:42:27.448978 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.448951 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" podUID="e618c702-d336-43d9-8857-430b4fa5b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 23:42:27.542530 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.542496 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:27.739899 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.738791 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:27.844046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.844014 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-xks-ca\") pod \"85e518bd-b5ec-4504-85fa-8c012807bbe0\" (UID: \"85e518bd-b5ec-4504-85fa-8c012807bbe0\") " Oct 30 23:42:27.844593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.844568 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-openshift-ca\") pod \"85e518bd-b5ec-4504-85fa-8c012807bbe0\" (UID: \"85e518bd-b5ec-4504-85fa-8c012807bbe0\") " Oct 30 23:42:27.845100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.845077 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e518bd-b5ec-4504-85fa-8c012807bbe0-kube-api-access-xs5zl\" (UniqueName: \"kubernetes.io/projected/85e518bd-b5ec-4504-85fa-8c012807bbe0-kube-api-access-xs5zl\") pod \"85e518bd-b5ec-4504-85fa-8c012807bbe0\" (UID: \"85e518bd-b5ec-4504-85fa-8c012807bbe0\") " Oct 30 23:42:27.845558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.845128 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-openshift-ca" pod "85e518bd-b5ec-4504-85fa-8c012807bbe0" (UID: "85e518bd-b5ec-4504-85fa-8c012807bbe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:27.846115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.846050 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:27.848200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.848175 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-xks-ca" pod "85e518bd-b5ec-4504-85fa-8c012807bbe0" (UID: "85e518bd-b5ec-4504-85fa-8c012807bbe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:27.848633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.848602 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e518bd-b5ec-4504-85fa-8c012807bbe0-kube-api-access-xs5zl" pod "85e518bd-b5ec-4504-85fa-8c012807bbe0" (UID: "85e518bd-b5ec-4504-85fa-8c012807bbe0"). InnerVolumeSpecName "kube-api-access-xs5zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:27.948540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.948440 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs5zl\" (UniqueName: \"kubernetes.io/projected/85e518bd-b5ec-4504-85fa-8c012807bbe0-kube-api-access-xs5zl\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:27.948540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:27.948488 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e518bd-b5ec-4504-85fa-8c012807bbe0-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:28.343059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:28.343029 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" Oct 30 23:42:28.343706 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:28.343032 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2" event={"ID":"85e518bd-b5ec-4504-85fa-8c012807bbe0","Type":"ContainerDied","Data":"3c98ff09ee8ceef974f62834f15e348dba26ab5e58f50fc056e3a23cea27154b"} Oct 30 23:42:28.343777 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:28.343738 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c98ff09ee8ceef974f62834f15e348dba26ab5e58f50fc056e3a23cea27154b" Oct 30 23:42:28.345728 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:28.345698 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" event={"ID":"520a1cb3-77dc-47d6-add9-0c262b514835","Type":"ContainerStarted","Data":"9ae5523100a396fb2399cff176642e92f6fbf5f7c353059c3bacae37fa284b2d"} Oct 30 23:42:28.371631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:28.371589 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" podStartSLOduration=2.371576608 podStartE2EDuration="2.371576608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:28.367012907 +0000 UTC m=+1359.886014021" watchObservedRunningTime="2026-10-30 23:42:28.371576608 +0000 UTC m=+1359.890577698" Oct 30 23:42:28.447585 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:28.447551 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:42:28.447808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:28.447628 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" podUID="e618c702-d336-43d9-8857-430b4fa5b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 23:42:28.540928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:28.540891 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:42:28.541231 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:28.541186 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 23:42:29.368747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:29.368712 7570 generic.go:356] "Generic (PLEG): container finished" podID="520a1cb3-77dc-47d6-add9-0c262b514835" containerID="9ae5523100a396fb2399cff176642e92f6fbf5f7c353059c3bacae37fa284b2d" exitCode=0 Oct 30 23:42:29.369722 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:29.369696 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" event={"ID":"520a1cb3-77dc-47d6-add9-0c262b514835","Type":"ContainerDied","Data":"9ae5523100a396fb2399cff176642e92f6fbf5f7c353059c3bacae37fa284b2d"} Oct 30 23:42:29.459489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:29.454004 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:42:29.459489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:29.454084 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" podUID="e618c702-d336-43d9-8857-430b4fa5b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 23:42:29.538488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:29.536863 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:42:29.538488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:29.536923 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 23:42:29.538488 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:42:29.537429 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:42:30.444488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.437551 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh"] Oct 30 23:42:30.444488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.438816 7570 cpu_manager.go:414] "containerMap: removing container" podUID="85e518bd-b5ec-4504-85fa-8c012807bbe0" containerName="revoke-keys" Oct 30 23:42:30.444488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.438838 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e518bd-b5ec-4504-85fa-8c012807bbe0" containerName="revoke-keys" Oct 30 23:42:30.444488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.439038 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e518bd-b5ec-4504-85fa-8c012807bbe0" containerName="revoke-keys" Oct 30 23:42:30.447844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.447810 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:42:30.448001 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.447885 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" podUID="e618c702-d336-43d9-8857-430b4fa5b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 23:42:30.448567 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.448546 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:30.450753 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.450711 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh"] Oct 30 23:42:30.538152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.537716 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:42:30.538152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.537778 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 23:42:30.551286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.551250 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4tv\" (UniqueName: \"kubernetes.io/projected/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-kube-api-access-ks4tv\") pod \"maas-api-revoke-sub-0f5daa2f-9t8qh\" (UID: \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:30.551523 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.551505 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f5daa2f-9t8qh\" (UID: \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:30.551679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.551657 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f5daa2f-9t8qh\" (UID: \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:30.654881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.654768 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f5daa2f-9t8qh\" (UID: \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:30.654881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.654874 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f5daa2f-9t8qh\" (UID: \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:30.655115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.655035 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4tv\" (UniqueName: \"kubernetes.io/projected/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-kube-api-access-ks4tv\") pod \"maas-api-revoke-sub-0f5daa2f-9t8qh\" (UID: \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:30.655925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.655898 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f5daa2f-9t8qh\" (UID: \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:30.661226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.661200 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f5daa2f-9t8qh\" (UID: \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:30.671743 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.671475 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4tv\" (UniqueName: \"kubernetes.io/projected/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-kube-api-access-ks4tv\") pod \"maas-api-revoke-sub-0f5daa2f-9t8qh\" (UID: \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:30.752769 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.752743 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:30.770940 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.770912 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:30.943571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.939899 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-xks-ca\") pod \"520a1cb3-77dc-47d6-add9-0c262b514835\" (UID: \"520a1cb3-77dc-47d6-add9-0c262b514835\") " Oct 30 23:42:30.943571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.940069 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-openshift-ca\") pod \"520a1cb3-77dc-47d6-add9-0c262b514835\" (UID: \"520a1cb3-77dc-47d6-add9-0c262b514835\") " Oct 30 23:42:30.951531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.951103 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520a1cb3-77dc-47d6-add9-0c262b514835-kube-api-access-hgs5x\" (UniqueName: \"kubernetes.io/projected/520a1cb3-77dc-47d6-add9-0c262b514835-kube-api-access-hgs5x\") pod \"520a1cb3-77dc-47d6-add9-0c262b514835\" (UID: \"520a1cb3-77dc-47d6-add9-0c262b514835\") " Oct 30 23:42:30.967857 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.967819 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520a1cb3-77dc-47d6-add9-0c262b514835-kube-api-access-hgs5x" pod "520a1cb3-77dc-47d6-add9-0c262b514835" (UID: "520a1cb3-77dc-47d6-add9-0c262b514835"). InnerVolumeSpecName "kube-api-access-hgs5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:30.979634 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:30.979574 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-openshift-ca" pod "520a1cb3-77dc-47d6-add9-0c262b514835" (UID: "520a1cb3-77dc-47d6-add9-0c262b514835"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:31.054765 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.054723 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-xks-ca" pod "520a1cb3-77dc-47d6-add9-0c262b514835" (UID: "520a1cb3-77dc-47d6-add9-0c262b514835"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:31.064893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.064856 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:31.064893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.064883 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgs5x\" (UniqueName: \"kubernetes.io/projected/520a1cb3-77dc-47d6-add9-0c262b514835-kube-api-access-hgs5x\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:31.064893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.064894 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520a1cb3-77dc-47d6-add9-0c262b514835-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:31.354786 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.354751 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh"] Oct 30 23:42:31.355074 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:42:31.355013 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086ece5e_33b4_46d3_bef7_48e28cc5ad3e.slice/crio-2cc8191d9acba96202f4808fe0d5f9248caa2c8d36d110f536bbfc63c5571142 WatchSource:0}: Error finding container 2cc8191d9acba96202f4808fe0d5f9248caa2c8d36d110f536bbfc63c5571142: Status 404 returned error can't find the container with id 2cc8191d9acba96202f4808fe0d5f9248caa2c8d36d110f536bbfc63c5571142 Oct 30 23:42:31.407488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.385669 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" event={"ID":"520a1cb3-77dc-47d6-add9-0c262b514835","Type":"ContainerDied","Data":"0935314a78121e9c73ab62205f0feb566847c14b2db56bd8c96fd92b25dda434"} Oct 30 23:42:31.407488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.385731 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0935314a78121e9c73ab62205f0feb566847c14b2db56bd8c96fd92b25dda434" Oct 30 23:42:31.407488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.385797 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r" Oct 30 23:42:31.418542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.418429 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" event={"ID":"086ece5e-33b4-46d3-bef7-48e28cc5ad3e","Type":"ContainerStarted","Data":"2cc8191d9acba96202f4808fe0d5f9248caa2c8d36d110f536bbfc63c5571142"} Oct 30 23:42:31.447726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.447696 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:42:31.448123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.447758 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" podUID="e618c702-d336-43d9-8857-430b4fa5b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 23:42:31.537988 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.537937 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:42:31.538197 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:31.537996 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 23:42:32.448989 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:32.448946 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:42:32.449499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:32.449034 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" podUID="e618c702-d336-43d9-8857-430b4fa5b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 23:42:32.455057 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:32.455019 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" event={"ID":"086ece5e-33b4-46d3-bef7-48e28cc5ad3e","Type":"ContainerStarted","Data":"876521141f9b121dff6bd4728a57ef29a07060e89105a3ef7e51948fbaa6d73e"} Oct 30 23:42:32.482153 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:32.482095 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" podStartSLOduration=2.48208455 podStartE2EDuration="2.48208455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:32.481899865 +0000 UTC m=+1364.000900945" watchObservedRunningTime="2026-10-30 23:42:32.48208455 +0000 UTC m=+1364.001085640" Oct 30 23:42:32.537341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:32.537287 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:42:32.537598 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:32.537366 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 23:42:33.447238 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:33.447202 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 23:42:33.447448 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:33.447258 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" podUID="e618c702-d336-43d9-8857-430b4fa5b8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 23:42:33.466674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:33.466636 7570 generic.go:356] "Generic (PLEG): container finished" podID="086ece5e-33b4-46d3-bef7-48e28cc5ad3e" containerID="876521141f9b121dff6bd4728a57ef29a07060e89105a3ef7e51948fbaa6d73e" exitCode=0 Oct 30 23:42:33.467077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:33.466731 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" event={"ID":"086ece5e-33b4-46d3-bef7-48e28cc5ad3e","Type":"ContainerDied","Data":"876521141f9b121dff6bd4728a57ef29a07060e89105a3ef7e51948fbaa6d73e"} Oct 30 23:42:33.540153 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:33.536687 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:42:33.540153 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:33.536751 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 23:42:34.707115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.707078 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:34.718942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.713320 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:34.719381 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.719350 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:42:34.734609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.733408 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:34.841661 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.841628 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:34.845529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.845508 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:34.846890 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.846867 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:42:34.875413 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.875022 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-openshift-ca\") pod \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\" (UID: \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\") " Oct 30 23:42:34.875413 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.875116 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-kube-api-access-ks4tv\" (UniqueName: \"kubernetes.io/projected/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-kube-api-access-ks4tv\") pod \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\" (UID: \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\") " Oct 30 23:42:34.875413 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.875411 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-xks-ca\") pod \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\" (UID: \"086ece5e-33b4-46d3-bef7-48e28cc5ad3e\") " Oct 30 23:42:34.875788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.875483 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-openshift-ca" pod "086ece5e-33b4-46d3-bef7-48e28cc5ad3e" (UID: "086ece5e-33b4-46d3-bef7-48e28cc5ad3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:34.876407 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.876381 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:34.879662 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.879619 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-xks-ca" pod "086ece5e-33b4-46d3-bef7-48e28cc5ad3e" (UID: "086ece5e-33b4-46d3-bef7-48e28cc5ad3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:34.879931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.879904 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-kube-api-access-ks4tv" pod "086ece5e-33b4-46d3-bef7-48e28cc5ad3e" (UID: "086ece5e-33b4-46d3-bef7-48e28cc5ad3e"). InnerVolumeSpecName "kube-api-access-ks4tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:34.979210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.978965 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks4tv\" (UniqueName: \"kubernetes.io/projected/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-kube-api-access-ks4tv\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:34.979210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:34.978998 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086ece5e-33b4-46d3-bef7-48e28cc5ad3e-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:35.489689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:35.489659 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" Oct 30 23:42:35.489689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:35.489666 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh" event={"ID":"086ece5e-33b4-46d3-bef7-48e28cc5ad3e","Type":"ContainerDied","Data":"2cc8191d9acba96202f4808fe0d5f9248caa2c8d36d110f536bbfc63c5571142"} Oct 30 23:42:35.489959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:35.489722 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cc8191d9acba96202f4808fe0d5f9248caa2c8d36d110f536bbfc63c5571142" Oct 30 23:42:38.937949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.937916 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k"] Oct 30 23:42:38.938678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.938664 7570 cpu_manager.go:414] "containerMap: removing container" podUID="086ece5e-33b4-46d3-bef7-48e28cc5ad3e" containerName="revoke-keys" Oct 30 23:42:38.938724 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.938680 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086ece5e-33b4-46d3-bef7-48e28cc5ad3e" containerName="revoke-keys" Oct 30 23:42:38.938724 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.938702 7570 cpu_manager.go:414] "containerMap: removing container" podUID="520a1cb3-77dc-47d6-add9-0c262b514835" containerName="revoke-keys" Oct 30 23:42:38.938724 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.938707 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520a1cb3-77dc-47d6-add9-0c262b514835" containerName="revoke-keys" Oct 30 23:42:38.938818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.938804 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="086ece5e-33b4-46d3-bef7-48e28cc5ad3e" containerName="revoke-keys" Oct 30 23:42:38.938818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.938816 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="520a1cb3-77dc-47d6-add9-0c262b514835" containerName="revoke-keys" Oct 30 23:42:38.942551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.942533 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:38.947300 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.947259 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:42:38.947845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.947822 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:42:38.953032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.952990 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c880c983-g884k\" (UID: \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:38.953153 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.953066 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k"] Oct 30 23:42:38.953153 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.953095 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5crp\" (UniqueName: \"kubernetes.io/projected/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-kube-api-access-w5crp\") pod \"maas-api-revoke-sub-c880c983-g884k\" (UID: \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:38.953397 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:38.953375 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c880c983-g884k\" (UID: \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:39.054886 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:39.054852 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c880c983-g884k\" (UID: \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:39.055070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:39.054942 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5crp\" (UniqueName: \"kubernetes.io/projected/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-kube-api-access-w5crp\") pod \"maas-api-revoke-sub-c880c983-g884k\" (UID: \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:39.055128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:39.055111 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c880c983-g884k\" (UID: \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:39.055646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:39.055626 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c880c983-g884k\" (UID: \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:39.058051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:39.058021 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c880c983-g884k\" (UID: \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:39.068791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:39.068768 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5crp\" (UniqueName: \"kubernetes.io/projected/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-kube-api-access-w5crp\") pod \"maas-api-revoke-sub-c880c983-g884k\" (UID: \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:39.288535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:39.288423 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:39.589624 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:42:39.589591 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8bc5a9_5d04_46fc_bbcb_ba005613f2e0.slice/crio-b76a8b9f7009bd066764e40a4744397c00533b2067de56ef3e042b277546ed77 WatchSource:0}: Error finding container b76a8b9f7009bd066764e40a4744397c00533b2067de56ef3e042b277546ed77: Status 404 returned error can't find the container with id b76a8b9f7009bd066764e40a4744397c00533b2067de56ef3e042b277546ed77 Oct 30 23:42:39.590382 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:39.590361 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k"] Oct 30 23:42:40.665488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:40.662111 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" podStartSLOduration=2.662088447 podStartE2EDuration="2.662088447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:40.630713469 +0000 UTC m=+1372.149714560" watchObservedRunningTime="2026-10-30 23:42:40.662088447 +0000 UTC m=+1372.181089539" Oct 30 23:42:40.748613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:40.748574 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" event={"ID":"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0","Type":"ContainerStarted","Data":"f39c679031f9aafb52ab04d1363de081f4a29a2abb51f84b5d35d53eb1335b15"} Oct 30 23:42:40.748843 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:40.748820 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" event={"ID":"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0","Type":"ContainerStarted","Data":"b76a8b9f7009bd066764e40a4744397c00533b2067de56ef3e042b277546ed77"} Oct 30 23:42:41.355487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:41.354562 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:41.355716 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:41.355684 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:41.669436 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:41.669312 7570 generic.go:356] "Generic (PLEG): container finished" podID="9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0" containerID="f39c679031f9aafb52ab04d1363de081f4a29a2abb51f84b5d35d53eb1335b15" exitCode=0 Oct 30 23:42:41.669926 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:41.669533 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" event={"ID":"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0","Type":"ContainerDied","Data":"f39c679031f9aafb52ab04d1363de081f4a29a2abb51f84b5d35d53eb1335b15"} Oct 30 23:42:41.779827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:41.767717 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:41.857488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:41.856942 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:42.113694 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.113646 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:42.154760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.132344 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:42:42.189487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.188753 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:42:42.197489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.190343 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:42.210834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.210594 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:42:42.354489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.346698 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:42.400661 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.391557 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:42.426936 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.412130 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:42.431486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.423605 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:42:42.457490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.423736 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:42:42.457969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.427410 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:42:42.458225 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.458206 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:42:42.669934 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.669815 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:42.774738 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.774672 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:42.905497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.831743 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:42:42.940524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.914663 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:42:42.940524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:42.915006 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:42:43.213540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.177772 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:43.236523 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.231963 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:42:43.262492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.197935 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:43.262492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.208215 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:43.262770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.262565 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:42:43.262770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.262645 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:42:43.262952 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.262916 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:43.273738 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.273705 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:42:43.300863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.208179 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:42:43.323354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.322432 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:43.342778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.342565 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:42:43.365486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.349544 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:43.365486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.351454 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:42:43.626261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.454284 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:43.626261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.459752 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:43.669310 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.628007 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:42:43.669310 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.628077 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:42:43.669310 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.460105 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:43.669310 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.460135 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:42:43.821488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.820643 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:43.933507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:43.932699 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:44.225753 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:44.225657 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:44.280507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:44.259666 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:42:44.280507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:44.262081 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:42:44.280507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:44.264511 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:44.344491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:44.330877 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:44.356697 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:44.356652 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:42:44.681695 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:44.659118 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:44.681695 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:44.660727 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:44.681695 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:44.662164 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:44.681695 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:44.662640 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:44.971579 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:44.969395 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:42:44.988724 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:44.988669 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:45.104191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.088572 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:45.235556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.229095 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:42:45.281307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.262196 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:45.281307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.262817 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:45.281307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.276645 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:45.348341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.333551 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:45.376437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.376400 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:45.483683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.483643 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:45.522925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.522779 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" event={"ID":"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0","Type":"ContainerDied","Data":"b76a8b9f7009bd066764e40a4744397c00533b2067de56ef3e042b277546ed77"} Oct 30 23:42:45.522925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.522841 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b76a8b9f7009bd066764e40a4744397c00533b2067de56ef3e042b277546ed77" Oct 30 23:42:45.608483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.580314 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-kube-api-access-w5crp\" (UniqueName: \"kubernetes.io/projected/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-kube-api-access-w5crp\") pod \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\" (UID: \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\") " Oct 30 23:42:45.608483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.580383 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-openshift-ca\") pod \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\" (UID: \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\") " Oct 30 23:42:45.608483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.580413 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-xks-ca\") pod \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\" (UID: \"9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0\") " Oct 30 23:42:45.608483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.600283 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-openshift-ca" pod "9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0" (UID: "9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:45.618920 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.618877 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-kube-api-access-w5crp" pod "9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0" (UID: "9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0"). InnerVolumeSpecName "kube-api-access-w5crp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:45.649761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.649717 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-xks-ca" pod "9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0" (UID: "9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:45.677493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.652177 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:42:45.677493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.659534 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:45.677493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.666668 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:42:45.677493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.667835 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:42:45.695491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.685598 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5crp\" (UniqueName: \"kubernetes.io/projected/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-kube-api-access-w5crp\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:45.695491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.685627 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:45.695491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.685647 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:45.721208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.721044 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:42:45.764514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.758513 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:42:45.764514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.758907 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:42:45.764514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.759007 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:42:45.764514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.759060 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:42:45.764514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.759109 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:42:45.764514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.759160 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:42:45.764514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.759208 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:42:45.764514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.759342 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:42:45.788494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.788395 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:42:45.789991 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:45.789905 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:42:46.530715 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:46.530685 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k" Oct 30 23:42:48.791387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:48.791355 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd"] Oct 30 23:42:48.792157 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:48.792139 7570 cpu_manager.go:414] "containerMap: removing container" podUID="9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0" containerName="revoke-keys" Oct 30 23:42:48.792218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:48.792157 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0" containerName="revoke-keys" Oct 30 23:42:48.792264 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:48.792254 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0" containerName="revoke-keys" Oct 30 23:42:48.797315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:48.797295 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:48.803434 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:48.803401 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd"] Oct 30 23:42:48.804434 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:48.804413 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:42:48.805586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:48.805556 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:42:48.955949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:48.955913 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61b1f042-dpmmd\" (UID: \"d57b9191-7694-4e32-acce-b6275d03180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:48.956216 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:48.956195 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61b1f042-dpmmd\" (UID: \"d57b9191-7694-4e32-acce-b6275d03180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:48.956300 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:48.956287 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n88l\" (UniqueName: \"kubernetes.io/projected/d57b9191-7694-4e32-acce-b6275d03180b-kube-api-access-9n88l\") pod \"maas-api-revoke-sub-61b1f042-dpmmd\" (UID: \"d57b9191-7694-4e32-acce-b6275d03180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:49.061212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:49.060834 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n88l\" (UniqueName: \"kubernetes.io/projected/d57b9191-7694-4e32-acce-b6275d03180b-kube-api-access-9n88l\") pod \"maas-api-revoke-sub-61b1f042-dpmmd\" (UID: \"d57b9191-7694-4e32-acce-b6275d03180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:49.068048 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:49.067999 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61b1f042-dpmmd\" (UID: \"d57b9191-7694-4e32-acce-b6275d03180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:49.068436 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:49.068407 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61b1f042-dpmmd\" (UID: \"d57b9191-7694-4e32-acce-b6275d03180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:49.070074 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:49.070003 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61b1f042-dpmmd\" (UID: \"d57b9191-7694-4e32-acce-b6275d03180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:49.080067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:49.080035 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61b1f042-dpmmd\" (UID: \"d57b9191-7694-4e32-acce-b6275d03180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:49.084213 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:49.084178 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n88l\" (UniqueName: \"kubernetes.io/projected/d57b9191-7694-4e32-acce-b6275d03180b-kube-api-access-9n88l\") pod \"maas-api-revoke-sub-61b1f042-dpmmd\" (UID: \"d57b9191-7694-4e32-acce-b6275d03180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:49.117230 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:49.117197 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:49.413791 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:42:49.413754 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57b9191_7694_4e32_acce_b6275d03180b.slice/crio-3d11eebe13307e54d20b013901c617d88207947c925c4c791ea16b8a11406ac5 WatchSource:0}: Error finding container 3d11eebe13307e54d20b013901c617d88207947c925c4c791ea16b8a11406ac5: Status 404 returned error can't find the container with id 3d11eebe13307e54d20b013901c617d88207947c925c4c791ea16b8a11406ac5 Oct 30 23:42:49.414266 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:49.414244 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd"] Oct 30 23:42:49.562088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:49.562046 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" event={"ID":"d57b9191-7694-4e32-acce-b6275d03180b","Type":"ContainerStarted","Data":"3d11eebe13307e54d20b013901c617d88207947c925c4c791ea16b8a11406ac5"} Oct 30 23:42:50.573521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:50.573413 7570 generic.go:356] "Generic (PLEG): container finished" podID="d57b9191-7694-4e32-acce-b6275d03180b" containerID="39ae65978ca64880fb0425e946c9df7d7000b3f3602ebd44712fca370c145ea4" exitCode=0 Oct 30 23:42:50.574011 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:50.573508 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" event={"ID":"d57b9191-7694-4e32-acce-b6275d03180b","Type":"ContainerDied","Data":"39ae65978ca64880fb0425e946c9df7d7000b3f3602ebd44712fca370c145ea4"} Oct 30 23:42:51.795968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:51.792624 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:51.917965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:51.917881 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57b9191-7694-4e32-acce-b6275d03180b-kube-api-access-9n88l\" (UniqueName: \"kubernetes.io/projected/d57b9191-7694-4e32-acce-b6275d03180b-kube-api-access-9n88l\") pod \"d57b9191-7694-4e32-acce-b6275d03180b\" (UID: \"d57b9191-7694-4e32-acce-b6275d03180b\") " Oct 30 23:42:51.918173 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:51.918158 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-xks-ca\") pod \"d57b9191-7694-4e32-acce-b6275d03180b\" (UID: \"d57b9191-7694-4e32-acce-b6275d03180b\") " Oct 30 23:42:51.918227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:51.918198 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-openshift-ca\") pod \"d57b9191-7694-4e32-acce-b6275d03180b\" (UID: \"d57b9191-7694-4e32-acce-b6275d03180b\") " Oct 30 23:42:51.918723 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:51.918690 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-openshift-ca" pod "d57b9191-7694-4e32-acce-b6275d03180b" (UID: "d57b9191-7694-4e32-acce-b6275d03180b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:51.918973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:51.918952 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:51.920788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:51.920766 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57b9191-7694-4e32-acce-b6275d03180b-kube-api-access-9n88l" pod "d57b9191-7694-4e32-acce-b6275d03180b" (UID: "d57b9191-7694-4e32-acce-b6275d03180b"). InnerVolumeSpecName "kube-api-access-9n88l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:51.921065 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:51.921029 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-xks-ca" pod "d57b9191-7694-4e32-acce-b6275d03180b" (UID: "d57b9191-7694-4e32-acce-b6275d03180b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:52.020928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:52.020898 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57b9191-7694-4e32-acce-b6275d03180b-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:52.020928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:52.020927 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n88l\" (UniqueName: \"kubernetes.io/projected/d57b9191-7694-4e32-acce-b6275d03180b-kube-api-access-9n88l\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:52.591473 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:52.591430 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" Oct 30 23:42:52.591692 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:52.591510 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd" event={"ID":"d57b9191-7694-4e32-acce-b6275d03180b","Type":"ContainerDied","Data":"3d11eebe13307e54d20b013901c617d88207947c925c4c791ea16b8a11406ac5"} Oct 30 23:42:52.591692 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:52.591571 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d11eebe13307e54d20b013901c617d88207947c925c4c791ea16b8a11406ac5" Oct 30 23:42:53.312479 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.312435 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv"] Oct 30 23:42:53.313190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.313175 7570 cpu_manager.go:414] "containerMap: removing container" podUID="d57b9191-7694-4e32-acce-b6275d03180b" containerName="revoke-keys" Oct 30 23:42:53.313230 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.313191 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57b9191-7694-4e32-acce-b6275d03180b" containerName="revoke-keys" Oct 30 23:42:53.313335 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.313326 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57b9191-7694-4e32-acce-b6275d03180b" containerName="revoke-keys" Oct 30 23:42:53.317376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.317361 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:53.322753 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.322723 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:42:53.324223 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.324202 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:42:53.325112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.325093 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv"] Oct 30 23:42:53.343766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.343736 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7jfj\" (UniqueName: \"kubernetes.io/projected/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-kube-api-access-n7jfj\") pod \"maas-api-revoke-sub-522c69d8-562nv\" (UID: \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:53.343943 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.343926 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-522c69d8-562nv\" (UID: \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:53.344042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.344026 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-522c69d8-562nv\" (UID: \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:53.445422 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.445384 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-522c69d8-562nv\" (UID: \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:53.445680 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.445574 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-522c69d8-562nv\" (UID: \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:53.445826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.445809 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7jfj\" (UniqueName: \"kubernetes.io/projected/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-kube-api-access-n7jfj\") pod \"maas-api-revoke-sub-522c69d8-562nv\" (UID: \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:53.447892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.447861 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-522c69d8-562nv\" (UID: \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:53.448722 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.448699 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-522c69d8-562nv\" (UID: \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:53.466420 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.466389 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7jfj\" (UniqueName: \"kubernetes.io/projected/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-kube-api-access-n7jfj\") pod \"maas-api-revoke-sub-522c69d8-562nv\" (UID: \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:53.636088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.636058 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:53.938076 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:53.938048 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv"] Oct 30 23:42:53.938208 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:42:53.938046 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea62cc2_1531_40fd_a104_9dcfb2d9e72f.slice/crio-634cdfb769cb6c2327c8deb1c995d2b009a89fdd3b5cabe900c0aa5bd140462f WatchSource:0}: Error finding container 634cdfb769cb6c2327c8deb1c995d2b009a89fdd3b5cabe900c0aa5bd140462f: Status 404 returned error can't find the container with id 634cdfb769cb6c2327c8deb1c995d2b009a89fdd3b5cabe900c0aa5bd140462f Oct 30 23:42:54.607764 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:54.607727 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" event={"ID":"aea62cc2-1531-40fd-a104-9dcfb2d9e72f","Type":"ContainerStarted","Data":"486be85973c7f9a10c7b572e17dacd1749f4123d0fe7af825eb3a404120fc545"} Oct 30 23:42:54.608213 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:54.607783 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" event={"ID":"aea62cc2-1531-40fd-a104-9dcfb2d9e72f","Type":"ContainerStarted","Data":"634cdfb769cb6c2327c8deb1c995d2b009a89fdd3b5cabe900c0aa5bd140462f"} Oct 30 23:42:54.629099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:54.629048 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" podStartSLOduration=1.629034418 podStartE2EDuration="1.629034418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:54.626094334 +0000 UTC m=+1386.145095426" watchObservedRunningTime="2026-10-30 23:42:54.629034418 +0000 UTC m=+1386.148035508" Oct 30 23:42:54.713892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:54.713852 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv"] Oct 30 23:42:54.714124 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:54.714097 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" podUID="2dd5d55c-9530-4b15-af81-8a87166097bb" containerName="istio-proxy" containerID="cri-o://8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf" gracePeriod=30 Oct 30 23:42:55.617036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.617002 7570 generic.go:356] "Generic (PLEG): container finished" podID="aea62cc2-1531-40fd-a104-9dcfb2d9e72f" containerID="486be85973c7f9a10c7b572e17dacd1749f4123d0fe7af825eb3a404120fc545" exitCode=0 Oct 30 23:42:55.619497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.617084 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" event={"ID":"aea62cc2-1531-40fd-a104-9dcfb2d9e72f","Type":"ContainerDied","Data":"486be85973c7f9a10c7b572e17dacd1749f4123d0fe7af825eb3a404120fc545"} Oct 30 23:42:55.795269 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.795230 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9"] Oct 30 23:42:55.801510 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.801447 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:55.807586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.807558 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ebbe654c-openshift-default-dockercfg-79cv9\"" Oct 30 23:42:55.814597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.814548 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9"] Oct 30 23:42:55.891789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.891693 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-istio-token\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:55.891789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.891755 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-certs\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:55.892043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.892021 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-socket\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:55.892264 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.892224 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-data\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:55.892633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.892605 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-envoy\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:55.892789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.892703 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xbw\" (UniqueName: \"kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-kube-api-access-g7xbw\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:55.892932 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.892904 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-credential-socket\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:55.892981 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.892964 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a5dec2-0378-450a-9942-4d824434ba36-istio-podinfo\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:55.893109 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.893093 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a5dec2-0378-450a-9942-4d824434ba36-istiod-ca-cert\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:55.995488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.995328 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-envoy\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:55.995488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:55.995389 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xbw\" (UniqueName: \"kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-kube-api-access-g7xbw\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.013485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.011588 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-credential-socket\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.013485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.011666 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a5dec2-0378-450a-9942-4d824434ba36-istio-podinfo\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.013485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.011838 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a5dec2-0378-450a-9942-4d824434ba36-istiod-ca-cert\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.013485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.011904 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-istio-token\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.013485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.011977 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-certs\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.013485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.012065 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-socket\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.013485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.012115 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-data\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.013485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.012657 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-data\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.013485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.012954 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-credential-socket\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.039490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.037190 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-envoy\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.046823 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.045418 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a5dec2-0378-450a-9942-4d824434ba36-istiod-ca-cert\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.046823 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.045796 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-certs\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.064639 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.052197 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-socket\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.064639 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.064104 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a5dec2-0378-450a-9942-4d824434ba36-istio-podinfo\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.083274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.083234 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xbw\" (UniqueName: \"kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-kube-api-access-g7xbw\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.110501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.107407 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-istio-token\") pod \"e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.128962 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.128930 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:56.438167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.438118 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9"] Oct 30 23:42:56.442698 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.442659 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:56.442818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.442744 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:56.442818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.442789 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:42:56.662868 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.662836 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" event={"ID":"33a5dec2-0378-450a-9942-4d824434ba36","Type":"ContainerStarted","Data":"fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147"} Oct 30 23:42:56.663529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.663452 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" event={"ID":"33a5dec2-0378-450a-9942-4d824434ba36","Type":"ContainerStarted","Data":"fb11cf87fbeac137d1d92b49d7d1a7b0831157a88801fdefbbb732dd4bba9c72"} Oct 30 23:42:56.886056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.886027 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:56.916208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.916152 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" podStartSLOduration=1.916136576 podStartE2EDuration="1.916136576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:56.692851134 +0000 UTC m=+1388.211852225" watchObservedRunningTime="2026-10-30 23:42:56.916136576 +0000 UTC m=+1388.435137666" Oct 30 23:42:56.940709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.940673 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-openshift-ca\") pod \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\" (UID: \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\") " Oct 30 23:42:56.940956 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.940903 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-kube-api-access-n7jfj\" (UniqueName: \"kubernetes.io/projected/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-kube-api-access-n7jfj\") pod \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\" (UID: \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\") " Oct 30 23:42:56.941050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.941027 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-xks-ca\") pod \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\" (UID: \"aea62cc2-1531-40fd-a104-9dcfb2d9e72f\") " Oct 30 23:42:56.942320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.942274 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-openshift-ca" pod "aea62cc2-1531-40fd-a104-9dcfb2d9e72f" (UID: "aea62cc2-1531-40fd-a104-9dcfb2d9e72f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:56.957411 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.957375 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-xks-ca" pod "aea62cc2-1531-40fd-a104-9dcfb2d9e72f" (UID: "aea62cc2-1531-40fd-a104-9dcfb2d9e72f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:56.957582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:56.957413 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-kube-api-access-n7jfj" pod "aea62cc2-1531-40fd-a104-9dcfb2d9e72f" (UID: "aea62cc2-1531-40fd-a104-9dcfb2d9e72f"). InnerVolumeSpecName "kube-api-access-n7jfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:57.043826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.043792 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:57.043826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.043825 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:57.044013 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.043836 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7jfj\" (UniqueName: \"kubernetes.io/projected/aea62cc2-1531-40fd-a104-9dcfb2d9e72f-kube-api-access-n7jfj\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:42:57.129352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.129315 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:57.139485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.136344 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:57.510033 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.509996 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9"] Oct 30 23:42:57.510865 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.510844 7570 cpu_manager.go:414] "containerMap: removing container" podUID="aea62cc2-1531-40fd-a104-9dcfb2d9e72f" containerName="revoke-keys" Oct 30 23:42:57.510865 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.510866 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea62cc2-1531-40fd-a104-9dcfb2d9e72f" containerName="revoke-keys" Oct 30 23:42:57.511052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.511014 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea62cc2-1531-40fd-a104-9dcfb2d9e72f" containerName="revoke-keys" Oct 30 23:42:57.515078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.515057 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:42:57.530838 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.530814 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9"] Oct 30 23:42:57.655213 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.655179 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1db63510-k62l9\" (UID: \"3687f3e8-fe18-460b-b448-95c82af92975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:42:57.655411 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.655373 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1db63510-k62l9\" (UID: \"3687f3e8-fe18-460b-b448-95c82af92975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:42:57.655582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.655564 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wq4t\" (UniqueName: \"kubernetes.io/projected/3687f3e8-fe18-460b-b448-95c82af92975-kube-api-access-5wq4t\") pod \"maas-api-revoke-sub-1db63510-k62l9\" (UID: \"3687f3e8-fe18-460b-b448-95c82af92975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:42:57.676655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.676550 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" event={"ID":"aea62cc2-1531-40fd-a104-9dcfb2d9e72f","Type":"ContainerDied","Data":"634cdfb769cb6c2327c8deb1c995d2b009a89fdd3b5cabe900c0aa5bd140462f"} Oct 30 23:42:57.676655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.676621 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="634cdfb769cb6c2327c8deb1c995d2b009a89fdd3b5cabe900c0aa5bd140462f" Oct 30 23:42:57.677250 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.677010 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:57.677250 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.677237 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv" Oct 30 23:42:57.678533 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.678508 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:42:57.758151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.758109 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1db63510-k62l9\" (UID: \"3687f3e8-fe18-460b-b448-95c82af92975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:42:57.759612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.759589 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1db63510-k62l9\" (UID: \"3687f3e8-fe18-460b-b448-95c82af92975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:42:57.761056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.760993 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1db63510-k62l9\" (UID: \"3687f3e8-fe18-460b-b448-95c82af92975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:42:57.761278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.761256 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wq4t\" (UniqueName: \"kubernetes.io/projected/3687f3e8-fe18-460b-b448-95c82af92975-kube-api-access-5wq4t\") pod \"maas-api-revoke-sub-1db63510-k62l9\" (UID: \"3687f3e8-fe18-460b-b448-95c82af92975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:42:57.766412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.766381 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1db63510-k62l9\" (UID: \"3687f3e8-fe18-460b-b448-95c82af92975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:42:57.775349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.775326 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wq4t\" (UniqueName: \"kubernetes.io/projected/3687f3e8-fe18-460b-b448-95c82af92975-kube-api-access-5wq4t\") pod \"maas-api-revoke-sub-1db63510-k62l9\" (UID: \"3687f3e8-fe18-460b-b448-95c82af92975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:42:57.837817 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:57.837747 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:42:58.125786 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:42:58.125758 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3687f3e8_fe18_460b_b448_95c82af92975.slice/crio-55fe3850511a91d76a8501745dd59a52887a9ae673b6f11fb02becf2ff23c377 WatchSource:0}: Error finding container 55fe3850511a91d76a8501745dd59a52887a9ae673b6f11fb02becf2ff23c377: Status 404 returned error can't find the container with id 55fe3850511a91d76a8501745dd59a52887a9ae673b6f11fb02becf2ff23c377 Oct 30 23:42:58.126880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:58.126851 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9"] Oct 30 23:42:58.691427 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:58.691395 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" event={"ID":"3687f3e8-fe18-460b-b448-95c82af92975","Type":"ContainerStarted","Data":"6e6e36bf126f356a280d07b7eec337ee6e6a4f1456d14fd41ac96c86f261d398"} Oct 30 23:42:58.691945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:58.691449 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" event={"ID":"3687f3e8-fe18-460b-b448-95c82af92975","Type":"ContainerStarted","Data":"55fe3850511a91d76a8501745dd59a52887a9ae673b6f11fb02becf2ff23c377"} Oct 30 23:42:58.750326 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:58.750060 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" podStartSLOduration=1.750045132 podStartE2EDuration="1.750045132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:58.749265132 +0000 UTC m=+1390.268266228" watchObservedRunningTime="2026-10-30 23:42:58.750045132 +0000 UTC m=+1390.269046260" Oct 30 23:42:59.417270 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:42:59.417226 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:42:59.855830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:59.855792 7570 generic.go:356] "Generic (PLEG): container finished" podID="3687f3e8-fe18-460b-b448-95c82af92975" containerID="6e6e36bf126f356a280d07b7eec337ee6e6a4f1456d14fd41ac96c86f261d398" exitCode=0 Oct 30 23:42:59.883943 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:42:59.877773 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" event={"ID":"3687f3e8-fe18-460b-b448-95c82af92975","Type":"ContainerDied","Data":"6e6e36bf126f356a280d07b7eec337ee6e6a4f1456d14fd41ac96c86f261d398"} Oct 30 23:43:00.125265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.125238 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:43:00.234637 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.234606 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-data\") pod \"2dd5d55c-9530-4b15-af81-8a87166097bb\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " Oct 30 23:43:00.234812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.234675 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd5d55c-9530-4b15-af81-8a87166097bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd5d55c-9530-4b15-af81-8a87166097bb-istiod-ca-cert\") pod \"2dd5d55c-9530-4b15-af81-8a87166097bb\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " Oct 30 23:43:00.234812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.234718 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-credential-socket\") pod \"2dd5d55c-9530-4b15-af81-8a87166097bb\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " Oct 30 23:43:00.234812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.234794 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-token\" (UniqueName: \"kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-token\") pod \"2dd5d55c-9530-4b15-af81-8a87166097bb\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " Oct 30 23:43:00.234980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.234864 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-socket\") pod \"2dd5d55c-9530-4b15-af81-8a87166097bb\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " Oct 30 23:43:00.234980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.234896 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-data" pod "2dd5d55c-9530-4b15-af81-8a87166097bb" (UID: "2dd5d55c-9530-4b15-af81-8a87166097bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:43:00.235086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.235000 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-envoy\") pod \"2dd5d55c-9530-4b15-af81-8a87166097bb\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " Oct 30 23:43:00.235086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.235059 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-kube-api-access-plxln\" (UniqueName: \"kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-kube-api-access-plxln\") pod \"2dd5d55c-9530-4b15-af81-8a87166097bb\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " Oct 30 23:43:00.235193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.235098 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-credential-socket" pod "2dd5d55c-9530-4b15-af81-8a87166097bb" (UID: "2dd5d55c-9530-4b15-af81-8a87166097bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:43:00.235193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.235143 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd5d55c-9530-4b15-af81-8a87166097bb-istiod-ca-cert" pod "2dd5d55c-9530-4b15-af81-8a87166097bb" (UID: "2dd5d55c-9530-4b15-af81-8a87166097bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:43:00.235193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.235156 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-podinfo\") pod \"2dd5d55c-9530-4b15-af81-8a87166097bb\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " Oct 30 23:43:00.235193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.235169 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-socket" pod "2dd5d55c-9530-4b15-af81-8a87166097bb" (UID: "2dd5d55c-9530-4b15-af81-8a87166097bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:43:00.235431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.235194 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-certs\") pod \"2dd5d55c-9530-4b15-af81-8a87166097bb\" (UID: \"2dd5d55c-9530-4b15-af81-8a87166097bb\") " Oct 30 23:43:00.235560 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.235525 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-certs" pod "2dd5d55c-9530-4b15-af81-8a87166097bb" (UID: "2dd5d55c-9530-4b15-af81-8a87166097bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:43:00.236305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.236276 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:00.236418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.236308 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:00.236418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.236324 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd5d55c-9530-4b15-af81-8a87166097bb-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:00.236418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.236340 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:00.236418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.236354 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:00.237582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.237528 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-kube-api-access-plxln" pod "2dd5d55c-9530-4b15-af81-8a87166097bb" (UID: "2dd5d55c-9530-4b15-af81-8a87166097bb"). InnerVolumeSpecName "kube-api-access-plxln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:43:00.237582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.237563 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-token" pod "2dd5d55c-9530-4b15-af81-8a87166097bb" (UID: "2dd5d55c-9530-4b15-af81-8a87166097bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:43:00.237956 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.237933 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-podinfo" pod "2dd5d55c-9530-4b15-af81-8a87166097bb" (UID: "2dd5d55c-9530-4b15-af81-8a87166097bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:43:00.237956 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.237936 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-envoy" pod "2dd5d55c-9530-4b15-af81-8a87166097bb" (UID: "2dd5d55c-9530-4b15-af81-8a87166097bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:43:00.337944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.337903 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:00.337944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.337937 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:00.337944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.337951 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd5d55c-9530-4b15-af81-8a87166097bb-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:00.338194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.337964 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plxln\" (UniqueName: \"kubernetes.io/projected/2dd5d55c-9530-4b15-af81-8a87166097bb-kube-api-access-plxln\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:00.887830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.887736 7570 generic.go:356] "Generic (PLEG): container finished" podID="2dd5d55c-9530-4b15-af81-8a87166097bb" containerID="8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf" exitCode=0 Oct 30 23:43:00.887830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.887803 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" Oct 30 23:43:00.888488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.887808 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" event={"ID":"2dd5d55c-9530-4b15-af81-8a87166097bb","Type":"ContainerDied","Data":"8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf"} Oct 30 23:43:00.888488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.887937 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv" event={"ID":"2dd5d55c-9530-4b15-af81-8a87166097bb","Type":"ContainerDied","Data":"50f4723b56bbc1c9fe4090df0857827442d9c2fe9095053c3b63474f19767e35"} Oct 30 23:43:00.888488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.887972 7570 scope.go:122] "RemoveContainer" containerID="8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf" Oct 30 23:43:00.903991 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.902973 7570 scope.go:122] "RemoveContainer" containerID="8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf" Oct 30 23:43:00.903991 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:43:00.903344 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf\": container with ID starting with 8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf not found: ID does not exist" containerID="8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf" Oct 30 23:43:00.903991 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.903377 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf"} err="failed to get container status \"8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf\": rpc error: code = NotFound desc = could not find container \"8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf\": container with ID starting with 8ef31acb8e6da41815fccf03537fe4fd3b4012a858885fbe55da3ba6d0a8cccf not found: ID does not exist" Oct 30 23:43:00.925404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.925367 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv"] Oct 30 23:43:00.931137 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:00.931109 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g8plv"] Oct 30 23:43:01.124769 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.124741 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:43:01.183837 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.183741 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-openshift-ca\") pod \"3687f3e8-fe18-460b-b448-95c82af92975\" (UID: \"3687f3e8-fe18-460b-b448-95c82af92975\") " Oct 30 23:43:01.184016 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.183899 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3687f3e8-fe18-460b-b448-95c82af92975-kube-api-access-5wq4t\" (UniqueName: \"kubernetes.io/projected/3687f3e8-fe18-460b-b448-95c82af92975-kube-api-access-5wq4t\") pod \"3687f3e8-fe18-460b-b448-95c82af92975\" (UID: \"3687f3e8-fe18-460b-b448-95c82af92975\") " Oct 30 23:43:01.184085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.184052 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-xks-ca\") pod \"3687f3e8-fe18-460b-b448-95c82af92975\" (UID: \"3687f3e8-fe18-460b-b448-95c82af92975\") " Oct 30 23:43:01.184206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.184176 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-openshift-ca" pod "3687f3e8-fe18-460b-b448-95c82af92975" (UID: "3687f3e8-fe18-460b-b448-95c82af92975"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:43:01.184775 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.184752 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:01.187084 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.187058 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-xks-ca" pod "3687f3e8-fe18-460b-b448-95c82af92975" (UID: "3687f3e8-fe18-460b-b448-95c82af92975"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:43:01.187212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.187186 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3687f3e8-fe18-460b-b448-95c82af92975-kube-api-access-5wq4t" pod "3687f3e8-fe18-460b-b448-95c82af92975" (UID: "3687f3e8-fe18-460b-b448-95c82af92975"). InnerVolumeSpecName "kube-api-access-5wq4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:43:01.286852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.286822 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wq4t\" (UniqueName: \"kubernetes.io/projected/3687f3e8-fe18-460b-b448-95c82af92975-kube-api-access-5wq4t\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:01.286852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.286850 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3687f3e8-fe18-460b-b448-95c82af92975-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:01.899562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.899519 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" event={"ID":"3687f3e8-fe18-460b-b448-95c82af92975","Type":"ContainerDied","Data":"55fe3850511a91d76a8501745dd59a52887a9ae673b6f11fb02becf2ff23c377"} Oct 30 23:43:01.900187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.899581 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55fe3850511a91d76a8501745dd59a52887a9ae673b6f11fb02becf2ff23c377" Oct 30 23:43:01.900187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:01.899531 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9" Oct 30 23:43:02.605641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:02.605610 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd5d55c-9530-4b15-af81-8a87166097bb" path="/var/lib/kubelet/pods/2dd5d55c-9530-4b15-af81-8a87166097bb/volumes" Oct 30 23:43:03.940284 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:03.940248 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd"] Oct 30 23:43:03.942808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:03.942780 7570 cpu_manager.go:414] "containerMap: removing container" podUID="2dd5d55c-9530-4b15-af81-8a87166097bb" containerName="istio-proxy" Oct 30 23:43:03.942808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:03.942808 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd5d55c-9530-4b15-af81-8a87166097bb" containerName="istio-proxy" Oct 30 23:43:03.943005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:03.942836 7570 cpu_manager.go:414] "containerMap: removing container" podUID="3687f3e8-fe18-460b-b448-95c82af92975" containerName="revoke-keys" Oct 30 23:43:03.943005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:03.942844 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3687f3e8-fe18-460b-b448-95c82af92975" containerName="revoke-keys" Oct 30 23:43:03.943311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:03.943285 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd5d55c-9530-4b15-af81-8a87166097bb" containerName="istio-proxy" Oct 30 23:43:03.943368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:03.943319 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3687f3e8-fe18-460b-b448-95c82af92975" containerName="revoke-keys" Oct 30 23:43:03.955689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:03.955663 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:03.967390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:03.966903 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d34ab3fd-openshift-default-dockercfg-ss8cz\"" Oct 30 23:43:03.974404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:03.974377 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd"] Oct 30 23:43:04.163631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.158887 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d7499f-a997-4d6e-9c62-57ce12871c43-istiod-ca-cert\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.163631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.158982 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-data\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.163631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.159081 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-socket\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.163631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.159165 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-certs\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.163631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.159191 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572t4\" (UniqueName: \"kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-kube-api-access-572t4\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.163631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.159274 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-podinfo\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.163631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.159331 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-credential-socket\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.163631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.159412 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-envoy\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.163631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.159648 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-token\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.261050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.260946 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-token\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.261050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.261029 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d7499f-a997-4d6e-9c62-57ce12871c43-istiod-ca-cert\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.261298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.261081 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-data\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.261298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.261152 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-socket\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.261298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.261219 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-certs\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.261298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.261243 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572t4\" (UniqueName: \"kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-kube-api-access-572t4\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.261891 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.261323 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-podinfo\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.261891 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.261371 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-credential-socket\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.261891 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.261445 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-envoy\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.262153 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.262133 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d7499f-a997-4d6e-9c62-57ce12871c43-istiod-ca-cert\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.262343 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.262320 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-socket\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.262454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.262429 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-credential-socket\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.262624 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.262604 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-certs\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.262698 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.262663 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-data\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.264625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.264595 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-podinfo\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.264837 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.264817 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-envoy\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.276703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.276677 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-token\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.276891 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.276867 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-572t4\" (UniqueName: \"kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-kube-api-access-572t4\") pod \"e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.276994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.276975 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:04.590287 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:43:04.590258 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d7499f_a997_4d6e_9c62_57ce12871c43.slice/crio-53d7ebb1384f1ee00c64076db01842e78fecfb6a2d9287b7a6677fd30866af19 WatchSource:0}: Error finding container 53d7ebb1384f1ee00c64076db01842e78fecfb6a2d9287b7a6677fd30866af19: Status 404 returned error can't find the container with id 53d7ebb1384f1ee00c64076db01842e78fecfb6a2d9287b7a6677fd30866af19 Oct 30 23:43:04.590416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.590378 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd"] Oct 30 23:43:04.592965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.592933 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:04.593036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.592998 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:04.593036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.593030 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:04.931684 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.931586 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" event={"ID":"a7d7499f-a997-4d6e-9c62-57ce12871c43","Type":"ContainerStarted","Data":"8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e"} Oct 30 23:43:04.931684 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.931646 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" event={"ID":"a7d7499f-a997-4d6e-9c62-57ce12871c43","Type":"ContainerStarted","Data":"53d7ebb1384f1ee00c64076db01842e78fecfb6a2d9287b7a6677fd30866af19"} Oct 30 23:43:04.991345 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:04.991274 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" podStartSLOduration=1.991261092 podStartE2EDuration="1.991261092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:04.988856194 +0000 UTC m=+1396.507857284" watchObservedRunningTime="2026-10-30 23:43:04.991261092 +0000 UTC m=+1396.510262181" Oct 30 23:43:05.278210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.278130 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:05.283913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.283883 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:05.529940 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.529859 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz"] Oct 30 23:43:05.534675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.534656 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:05.539366 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.539342 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:43:05.539679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.539474 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:43:05.544686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.544651 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz"] Oct 30 23:43:05.696114 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.696077 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f795a373-lxvrz\" (UID: \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:05.696297 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.696272 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f795a373-lxvrz\" (UID: \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:05.696523 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.696454 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fjg\" (UniqueName: \"kubernetes.io/projected/dac9cff3-e332-4a93-98a4-7ba14bd679f3-kube-api-access-m5fjg\") pod \"maas-api-revoke-sub-f795a373-lxvrz\" (UID: \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:05.797708 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.797624 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f795a373-lxvrz\" (UID: \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:05.797868 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.797770 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fjg\" (UniqueName: \"kubernetes.io/projected/dac9cff3-e332-4a93-98a4-7ba14bd679f3-kube-api-access-m5fjg\") pod \"maas-api-revoke-sub-f795a373-lxvrz\" (UID: \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:05.797925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.797874 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f795a373-lxvrz\" (UID: \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:05.798333 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.798310 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f795a373-lxvrz\" (UID: \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:05.800434 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.800411 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f795a373-lxvrz\" (UID: \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:05.813446 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.813420 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fjg\" (UniqueName: \"kubernetes.io/projected/dac9cff3-e332-4a93-98a4-7ba14bd679f3-kube-api-access-m5fjg\") pod \"maas-api-revoke-sub-f795a373-lxvrz\" (UID: \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:05.852855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.852821 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:05.955398 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.955366 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:05.958306 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:05.958284 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:43:06.172907 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:43:06.172866 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac9cff3_e332_4a93_98a4_7ba14bd679f3.slice/crio-10acb1b6b453fdac39064909002ac53db23ae0899059f04a8276d03a7920d3c2 WatchSource:0}: Error finding container 10acb1b6b453fdac39064909002ac53db23ae0899059f04a8276d03a7920d3c2: Status 404 returned error can't find the container with id 10acb1b6b453fdac39064909002ac53db23ae0899059f04a8276d03a7920d3c2 Oct 30 23:43:06.177201 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:06.177174 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz"] Oct 30 23:43:06.966425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:06.966337 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" event={"ID":"dac9cff3-e332-4a93-98a4-7ba14bd679f3","Type":"ContainerStarted","Data":"f4e91fc31d2b376736a04c73e73a1f18650434fe6d917af29a8dfe4e51814e52"} Oct 30 23:43:06.966425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:06.966395 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" event={"ID":"dac9cff3-e332-4a93-98a4-7ba14bd679f3","Type":"ContainerStarted","Data":"10acb1b6b453fdac39064909002ac53db23ae0899059f04a8276d03a7920d3c2"} Oct 30 23:43:07.978813 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:07.978766 7570 generic.go:356] "Generic (PLEG): container finished" podID="dac9cff3-e332-4a93-98a4-7ba14bd679f3" containerID="f4e91fc31d2b376736a04c73e73a1f18650434fe6d917af29a8dfe4e51814e52" exitCode=0 Oct 30 23:43:07.979235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:07.978822 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" event={"ID":"dac9cff3-e332-4a93-98a4-7ba14bd679f3","Type":"ContainerDied","Data":"f4e91fc31d2b376736a04c73e73a1f18650434fe6d917af29a8dfe4e51814e52"} Oct 30 23:43:08.370791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:08.370755 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7"] Oct 30 23:43:08.376905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:08.376880 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" Oct 30 23:43:08.381298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:08.381273 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d34ab3fd\"" Oct 30 23:43:08.384812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:08.384779 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d34ab3fd-dockercfg-tschw\"" Oct 30 23:43:08.388554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:08.388528 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7"] Oct 30 23:43:08.601889 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:08.600087 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvs6k\" (UniqueName: \"kubernetes.io/projected/e96b553f-6765-4476-a306-4c5702af33c5-kube-api-access-mvs6k\") pod \"payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7\" (UID: \"e96b553f-6765-4476-a306-4c5702af33c5\") " pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" Oct 30 23:43:08.601889 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:08.600211 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96b553f-6765-4476-a306-4c5702af33c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7\" (UID: \"e96b553f-6765-4476-a306-4c5702af33c5\") " pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" Oct 30 23:43:08.701726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:08.701630 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96b553f-6765-4476-a306-4c5702af33c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7\" (UID: \"e96b553f-6765-4476-a306-4c5702af33c5\") " pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" Oct 30 23:43:08.702067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:08.702044 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvs6k\" (UniqueName: \"kubernetes.io/projected/e96b553f-6765-4476-a306-4c5702af33c5-kube-api-access-mvs6k\") pod \"payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7\" (UID: \"e96b553f-6765-4476-a306-4c5702af33c5\") " pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" Oct 30 23:43:08.703234 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:08.703207 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96b553f-6765-4476-a306-4c5702af33c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7\" (UID: \"e96b553f-6765-4476-a306-4c5702af33c5\") " pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" Oct 30 23:43:08.716450 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:08.716417 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvs6k\" (UniqueName: \"kubernetes.io/projected/e96b553f-6765-4476-a306-4c5702af33c5-kube-api-access-mvs6k\") pod \"payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7\" (UID: \"e96b553f-6765-4476-a306-4c5702af33c5\") " pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" Oct 30 23:43:09.009285 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:09.009254 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" Oct 30 23:43:09.179062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:09.179034 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:09.219025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:09.218991 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-xks-ca\") pod \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\" (UID: \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\") " Oct 30 23:43:09.219226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:09.219080 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-openshift-ca\") pod \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\" (UID: \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\") " Oct 30 23:43:09.219226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:09.219154 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac9cff3-e332-4a93-98a4-7ba14bd679f3-kube-api-access-m5fjg\" (UniqueName: \"kubernetes.io/projected/dac9cff3-e332-4a93-98a4-7ba14bd679f3-kube-api-access-m5fjg\") pod \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\" (UID: \"dac9cff3-e332-4a93-98a4-7ba14bd679f3\") " Oct 30 23:43:09.219495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:09.219454 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-openshift-ca" pod "dac9cff3-e332-4a93-98a4-7ba14bd679f3" (UID: "dac9cff3-e332-4a93-98a4-7ba14bd679f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:43:09.220284 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:09.220255 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:09.222202 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:09.222178 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-xks-ca" pod "dac9cff3-e332-4a93-98a4-7ba14bd679f3" (UID: "dac9cff3-e332-4a93-98a4-7ba14bd679f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:43:09.222801 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:09.222775 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac9cff3-e332-4a93-98a4-7ba14bd679f3-kube-api-access-m5fjg" pod "dac9cff3-e332-4a93-98a4-7ba14bd679f3" (UID: "dac9cff3-e332-4a93-98a4-7ba14bd679f3"). InnerVolumeSpecName "kube-api-access-m5fjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:43:09.322281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:09.322189 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac9cff3-e332-4a93-98a4-7ba14bd679f3-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:09.322281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:09.322222 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5fjg\" (UniqueName: \"kubernetes.io/projected/dac9cff3-e332-4a93-98a4-7ba14bd679f3-kube-api-access-m5fjg\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:09.339432 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:43:09.339400 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96b553f_6765_4476_a306_4c5702af33c5.slice/crio-62b22fbb03471803a6189a22a11b7d8fba587d1c6b90dc965debcdf3f7741263 WatchSource:0}: Error finding container 62b22fbb03471803a6189a22a11b7d8fba587d1c6b90dc965debcdf3f7741263: Status 404 returned error can't find the container with id 62b22fbb03471803a6189a22a11b7d8fba587d1c6b90dc965debcdf3f7741263 Oct 30 23:43:09.339432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:09.339410 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7"] Oct 30 23:43:10.078559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:10.074689 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" event={"ID":"e96b553f-6765-4476-a306-4c5702af33c5","Type":"ContainerStarted","Data":"d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785"} Oct 30 23:43:10.078559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:10.074817 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" event={"ID":"e96b553f-6765-4476-a306-4c5702af33c5","Type":"ContainerStarted","Data":"62b22fbb03471803a6189a22a11b7d8fba587d1c6b90dc965debcdf3f7741263"} Oct 30 23:43:10.078559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:10.076601 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" Oct 30 23:43:10.079200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:10.078900 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" event={"ID":"dac9cff3-e332-4a93-98a4-7ba14bd679f3","Type":"ContainerDied","Data":"10acb1b6b453fdac39064909002ac53db23ae0899059f04a8276d03a7920d3c2"} Oct 30 23:43:10.079200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:10.079193 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10acb1b6b453fdac39064909002ac53db23ae0899059f04a8276d03a7920d3c2" Oct 30 23:43:10.163501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:10.080210 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz" Oct 30 23:43:10.593488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:10.593309 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" podStartSLOduration=2.5932882040000003 podStartE2EDuration="2.593288204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:10.514856792 +0000 UTC m=+1402.033857883" watchObservedRunningTime="2026-10-30 23:43:10.593288204 +0000 UTC m=+1402.112289298" Oct 30 23:43:10.668771 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:10.668722 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:10.687804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:10.684940 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:10.878549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:10.873399 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:10.945489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:10.943570 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:10.995488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:10.978975 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:11.286548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.286192 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:11.366972 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.366915 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:11.540582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.538937 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:11.557501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.553551 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:11.675508 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.674554 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:43:11.676042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.676015 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:43:11.685574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.685539 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:43:11.686599 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.686567 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:43:11.687175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.687149 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:43:11.700486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.688682 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:43:11.762419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.762371 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:11.951494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.881176 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:12.085758 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.085732 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:43:12.089935 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:11.992616 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:43:12.090421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.059375 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:43:12.106903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.106871 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:43:12.125553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.125513 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:43:12.362554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.343074 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:43:12.362554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.344531 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:43:12.459489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.451611 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:43:12.459489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.452874 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:43:12.550494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.539306 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:12.562629 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.562592 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:43:12.596495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.564699 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:12.598340 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.598310 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:43:12.664409 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.664291 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:12.678706 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.678647 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:12.693500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.689724 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:12.700495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.698930 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:12.721591 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.704805 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:43:12.838947 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:12.838903 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:13.117725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.117565 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:13.118235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.118108 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:43:13.118611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.118576 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:13.366489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.366059 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:13.367031 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.366878 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 23:43:13.459548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.439564 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:43:13.459548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.439635 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:13.459548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.448173 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:43:13.459548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.448423 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:43:13.459548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.448658 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:43:13.460064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.460025 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:13.551256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.551217 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:13.609216 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.609160 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:13.689523 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.688602 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:43:13.736704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.694255 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:13.788172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.706616 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:13.789386 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.732624 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:43:13.874498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:13.863357 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:43:14.144491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:14.127031 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:43:14.170490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:14.127203 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:14.171795 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:14.171759 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:43:14.376616 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:14.376559 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:43:14.573551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:14.572584 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:43:14.577489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:14.575032 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:14.626706 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:14.618304 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:14.749488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:14.748552 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:14.829554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:14.825817 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:43:14.865523 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:14.825895 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:14.911491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:14.909214 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:43:14.911491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:14.826300 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:43:15.068491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.064593 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:15.188571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.136508 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:43:15.188571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.136607 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:15.188571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.148801 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:43:15.196712 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.196673 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:43:15.200487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.196940 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:15.233072 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.220095 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:43:15.266511 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.249039 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:43:15.399537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.389375 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:43:15.594131 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.594076 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:15.625126 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.594400 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:43:15.651551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.622145 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:15.678031 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.677988 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:43:15.678421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.678384 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:15.832490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.828644 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:43:15.892575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.863550 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:43:15.892575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.865300 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:43:15.912702 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.912653 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:15.915720 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:15.915679 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:16.166559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.135081 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:16.243487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.151042 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:16.288488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.250356 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:43:16.288488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.251150 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:43:16.454796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.454668 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:16.635621 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.635569 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:16.652597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.652552 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:16.661941 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.661565 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:16.837604 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.837558 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:43:16.865618 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.865569 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:43:16.866245 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.866208 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:16.922776 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.922729 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:16.940227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:16.940184 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:43:17.245622 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.244557 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:17.251069 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.250544 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:43:17.251754 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.251312 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:43:17.771499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.765836 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:43:17.885558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.790044 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:17.885558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.790985 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:43:17.885558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.791761 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:43:17.885558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.821412 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 30 23:43:17.885558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.844563 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:17.885558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.847690 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:17.902579 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.886139 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:17.902579 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.886532 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:43:17.967509 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.967446 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:43:18.007506 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:17.969138 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:18.089031 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.088966 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:18.260632 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.260583 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:18.261976 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.261942 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:18.283122 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.283077 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:18.770643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.768183 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:18.770643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.768436 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:43:18.798717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.798666 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:18.831644 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.807223 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:18.831644 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.807289 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:43:18.848512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.847594 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:18.901789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.901761 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:43:18.915496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.912205 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:18.950487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.945720 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:18.950487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.945961 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:43:19.011098 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.999024 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:19.012645 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:18.998544 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:19.064237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:19.064156 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:43:19.065948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:19.065923 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:43:19.066579 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:19.066562 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:43:19.068906 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:19.068893 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:43:19.069791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:19.069644 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:43:19.071662 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:19.071403 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:43:19.077854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:19.077836 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:43:19.136039 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:19.134923 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:43:19.157988 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:19.157956 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:43:19.167139 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:19.167106 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:43:19.175499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:19.175477 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:43:20.632137 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.632101 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r"] Oct 30 23:43:20.632855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.632840 7570 cpu_manager.go:414] "containerMap: removing container" podUID="dac9cff3-e332-4a93-98a4-7ba14bd679f3" containerName="revoke-keys" Oct 30 23:43:20.632896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.632857 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac9cff3-e332-4a93-98a4-7ba14bd679f3" containerName="revoke-keys" Oct 30 23:43:20.633000 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.632991 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac9cff3-e332-4a93-98a4-7ba14bd679f3" containerName="revoke-keys" Oct 30 23:43:20.638036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.638018 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:20.642479 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.642439 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:43:20.644671 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.644648 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:43:20.646100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.646072 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r"] Oct 30 23:43:20.714966 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.714929 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3efba4c9-fkh9r\" (UID: \"99d6813e-e082-4555-a067-2a7b062a60ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:20.715244 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.715219 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srqdf\" (UniqueName: \"kubernetes.io/projected/99d6813e-e082-4555-a067-2a7b062a60ad-kube-api-access-srqdf\") pod \"maas-api-revoke-sub-3efba4c9-fkh9r\" (UID: \"99d6813e-e082-4555-a067-2a7b062a60ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:20.715338 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.715324 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3efba4c9-fkh9r\" (UID: \"99d6813e-e082-4555-a067-2a7b062a60ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:20.816804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.816765 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srqdf\" (UniqueName: \"kubernetes.io/projected/99d6813e-e082-4555-a067-2a7b062a60ad-kube-api-access-srqdf\") pod \"maas-api-revoke-sub-3efba4c9-fkh9r\" (UID: \"99d6813e-e082-4555-a067-2a7b062a60ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:20.816982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.816858 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3efba4c9-fkh9r\" (UID: \"99d6813e-e082-4555-a067-2a7b062a60ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:20.816982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.816940 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3efba4c9-fkh9r\" (UID: \"99d6813e-e082-4555-a067-2a7b062a60ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:20.817836 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.817806 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3efba4c9-fkh9r\" (UID: \"99d6813e-e082-4555-a067-2a7b062a60ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:20.819593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.819568 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3efba4c9-fkh9r\" (UID: \"99d6813e-e082-4555-a067-2a7b062a60ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:20.831543 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.831519 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srqdf\" (UniqueName: \"kubernetes.io/projected/99d6813e-e082-4555-a067-2a7b062a60ad-kube-api-access-srqdf\") pod \"maas-api-revoke-sub-3efba4c9-fkh9r\" (UID: \"99d6813e-e082-4555-a067-2a7b062a60ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:20.956345 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:20.956258 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:21.255050 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:43:21.255014 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d6813e_e082_4555_a067_2a7b062a60ad.slice/crio-7fda8c96ff99429667027503e82db539f116a0e4372d1b174855736ee5c329f5 WatchSource:0}: Error finding container 7fda8c96ff99429667027503e82db539f116a0e4372d1b174855736ee5c329f5: Status 404 returned error can't find the container with id 7fda8c96ff99429667027503e82db539f116a0e4372d1b174855736ee5c329f5 Oct 30 23:43:21.257821 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:21.257798 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r"] Oct 30 23:43:21.540804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:21.540660 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" event={"ID":"99d6813e-e082-4555-a067-2a7b062a60ad","Type":"ContainerStarted","Data":"7fda8c96ff99429667027503e82db539f116a0e4372d1b174855736ee5c329f5"} Oct 30 23:43:22.455808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:22.455781 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" Oct 30 23:43:22.551360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:22.551253 7570 generic.go:356] "Generic (PLEG): container finished" podID="99d6813e-e082-4555-a067-2a7b062a60ad" containerID="4de508c96450866d11d19194631173a6e4839f71f9ba4cb506c7376a37938698" exitCode=0 Oct 30 23:43:22.551360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:22.551337 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" event={"ID":"99d6813e-e082-4555-a067-2a7b062a60ad","Type":"ContainerDied","Data":"4de508c96450866d11d19194631173a6e4839f71f9ba4cb506c7376a37938698"} Oct 30 23:43:23.744977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:23.744949 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:23.845658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:23.845627 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d6813e-e082-4555-a067-2a7b062a60ad-kube-api-access-srqdf\" (UniqueName: \"kubernetes.io/projected/99d6813e-e082-4555-a067-2a7b062a60ad-kube-api-access-srqdf\") pod \"99d6813e-e082-4555-a067-2a7b062a60ad\" (UID: \"99d6813e-e082-4555-a067-2a7b062a60ad\") " Oct 30 23:43:23.845844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:23.845684 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-xks-ca\") pod \"99d6813e-e082-4555-a067-2a7b062a60ad\" (UID: \"99d6813e-e082-4555-a067-2a7b062a60ad\") " Oct 30 23:43:23.845844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:23.845812 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-openshift-ca\") pod \"99d6813e-e082-4555-a067-2a7b062a60ad\" (UID: \"99d6813e-e082-4555-a067-2a7b062a60ad\") " Oct 30 23:43:23.846192 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:23.846156 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-openshift-ca" pod "99d6813e-e082-4555-a067-2a7b062a60ad" (UID: "99d6813e-e082-4555-a067-2a7b062a60ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:43:23.846515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:23.846403 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:23.848020 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:23.847997 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d6813e-e082-4555-a067-2a7b062a60ad-kube-api-access-srqdf" pod "99d6813e-e082-4555-a067-2a7b062a60ad" (UID: "99d6813e-e082-4555-a067-2a7b062a60ad"). InnerVolumeSpecName "kube-api-access-srqdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:43:23.848103 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:23.848083 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-xks-ca" pod "99d6813e-e082-4555-a067-2a7b062a60ad" (UID: "99d6813e-e082-4555-a067-2a7b062a60ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:43:23.948276 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:23.948241 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srqdf\" (UniqueName: \"kubernetes.io/projected/99d6813e-e082-4555-a067-2a7b062a60ad-kube-api-access-srqdf\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:23.948276 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:23.948273 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d6813e-e082-4555-a067-2a7b062a60ad-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:24.582570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:24.580600 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" event={"ID":"99d6813e-e082-4555-a067-2a7b062a60ad","Type":"ContainerDied","Data":"7fda8c96ff99429667027503e82db539f116a0e4372d1b174855736ee5c329f5"} Oct 30 23:43:24.582570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:24.580674 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fda8c96ff99429667027503e82db539f116a0e4372d1b174855736ee5c329f5" Oct 30 23:43:24.582570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:24.580757 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r" Oct 30 23:43:25.647909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.647842 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc"] Oct 30 23:43:25.648749 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.648728 7570 cpu_manager.go:414] "containerMap: removing container" podUID="99d6813e-e082-4555-a067-2a7b062a60ad" containerName="revoke-keys" Oct 30 23:43:25.648749 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.648749 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d6813e-e082-4555-a067-2a7b062a60ad" containerName="revoke-keys" Oct 30 23:43:25.648916 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.648899 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d6813e-e082-4555-a067-2a7b062a60ad" containerName="revoke-keys" Oct 30 23:43:25.653314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.653289 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" Oct 30 23:43:25.670953 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.670927 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1dd98b91\"" Oct 30 23:43:25.671113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.670952 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1dd98b91-dockercfg-2lc7x\"" Oct 30 23:43:25.680504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.680428 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc"] Oct 30 23:43:25.729487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.710821 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27pqc\" (UniqueName: \"kubernetes.io/projected/251680aa-f28b-4c53-be4f-47b0a6de4cef-kube-api-access-27pqc\") pod \"payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc\" (UID: \"251680aa-f28b-4c53-be4f-47b0a6de4cef\") " pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" Oct 30 23:43:25.729487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.711059 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251680aa-f28b-4c53-be4f-47b0a6de4cef-plugins-config-volume\") pod \"payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc\" (UID: \"251680aa-f28b-4c53-be4f-47b0a6de4cef\") " pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" Oct 30 23:43:25.841628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.823001 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27pqc\" (UniqueName: \"kubernetes.io/projected/251680aa-f28b-4c53-be4f-47b0a6de4cef-kube-api-access-27pqc\") pod \"payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc\" (UID: \"251680aa-f28b-4c53-be4f-47b0a6de4cef\") " pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" Oct 30 23:43:25.843545 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.843510 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251680aa-f28b-4c53-be4f-47b0a6de4cef-plugins-config-volume\") pod \"payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc\" (UID: \"251680aa-f28b-4c53-be4f-47b0a6de4cef\") " pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" Oct 30 23:43:25.846216 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.846185 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251680aa-f28b-4c53-be4f-47b0a6de4cef-plugins-config-volume\") pod \"payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc\" (UID: \"251680aa-f28b-4c53-be4f-47b0a6de4cef\") " pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" Oct 30 23:43:25.920908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.920826 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27pqc\" (UniqueName: \"kubernetes.io/projected/251680aa-f28b-4c53-be4f-47b0a6de4cef-kube-api-access-27pqc\") pod \"payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc\" (UID: \"251680aa-f28b-4c53-be4f-47b0a6de4cef\") " pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" Oct 30 23:43:25.981051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:25.981013 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" Oct 30 23:43:26.301869 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:43:26.301832 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251680aa_f28b_4c53_be4f_47b0a6de4cef.slice/crio-0ecb8f704d2e57a79c17b92b5ce94375954338ae722944a29fbecce0e7b0318d WatchSource:0}: Error finding container 0ecb8f704d2e57a79c17b92b5ce94375954338ae722944a29fbecce0e7b0318d: Status 404 returned error can't find the container with id 0ecb8f704d2e57a79c17b92b5ce94375954338ae722944a29fbecce0e7b0318d Oct 30 23:43:26.302056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.302036 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc"] Oct 30 23:43:26.408302 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.408272 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr"] Oct 30 23:43:26.413530 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.413504 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.419512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.418348 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d34ab3fd-serving-cert\"" Oct 30 23:43:26.423813 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.423652 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr"] Oct 30 23:43:26.453169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.453136 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-maas-api-tls\") pod \"maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.453291 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.453213 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc9jw\" (UniqueName: \"kubernetes.io/projected/17a6cbd6-0d33-4498-b390-ee202481a578-kube-api-access-jc9jw\") pod \"maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.453345 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.453311 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a6cbd6-0d33-4498-b390-ee202481a578-openshift-service-ca\") pod \"maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.453499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.453486 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-metrics-certs\") pod \"maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.555265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.555174 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-maas-api-tls\") pod \"maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.555265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.555226 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc9jw\" (UniqueName: \"kubernetes.io/projected/17a6cbd6-0d33-4498-b390-ee202481a578-kube-api-access-jc9jw\") pod \"maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.555536 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.555327 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a6cbd6-0d33-4498-b390-ee202481a578-openshift-service-ca\") pod \"maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.555633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.555614 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-metrics-certs\") pod \"maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.556430 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.556182 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a6cbd6-0d33-4498-b390-ee202481a578-openshift-service-ca\") pod \"maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.558417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.558388 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-maas-api-tls\") pod \"maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.558880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.558789 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-metrics-certs\") pod \"maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.585195 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.585133 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc9jw\" (UniqueName: \"kubernetes.io/projected/17a6cbd6-0d33-4498-b390-ee202481a578-kube-api-access-jc9jw\") pod \"maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.608118 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.608090 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" Oct 30 23:43:26.608292 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.608137 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" event={"ID":"251680aa-f28b-4c53-be4f-47b0a6de4cef","Type":"ContainerStarted","Data":"89a65c061be5cdc9c2845077a37a9630744fab9938adf1cf555c12cf8d721d31"} Oct 30 23:43:26.608292 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.608157 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" event={"ID":"251680aa-f28b-4c53-be4f-47b0a6de4cef","Type":"ContainerStarted","Data":"0ecb8f704d2e57a79c17b92b5ce94375954338ae722944a29fbecce0e7b0318d"} Oct 30 23:43:26.638102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.638049 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" podStartSLOduration=1.63803546 podStartE2EDuration="1.63803546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:26.635672022 +0000 UTC m=+1418.154673112" watchObservedRunningTime="2026-10-30 23:43:26.63803546 +0000 UTC m=+1418.157036594" Oct 30 23:43:26.710096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.710061 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq"] Oct 30 23:43:26.715282 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.715260 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.728492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.728430 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq"] Oct 30 23:43:26.731354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.731325 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:26.784716 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.784682 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.784879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.784783 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.784879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.784819 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.784997 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.784949 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.784997 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.784983 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6474f60e-9d99-45e5-830d-4a668dbd41bb-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.785100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.785036 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4sgl\" (UniqueName: \"kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-kube-api-access-d4sgl\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.785178 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.785162 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.785254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.785240 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.785324 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.785311 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.897418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.897369 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.897726 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.897833 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.897887 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.897989 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.898050 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.898078 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.898131 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.898149 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6474f60e-9d99-45e5-830d-4a668dbd41bb-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.898182 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4sgl\" (UniqueName: \"kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-kube-api-access-d4sgl\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.898806 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.898818 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.899325 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.899492 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6474f60e-9d99-45e5-830d-4a668dbd41bb-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.902455 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.903592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.903318 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.912561 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.912509 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4sgl\" (UniqueName: \"kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-kube-api-access-d4sgl\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:26.913876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:26.913832 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:27.035761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:27.035725 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:27.094454 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:43:27.094382 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a6cbd6_0d33_4498_b390_ee202481a578.slice/crio-26ef0fa9eb4f3943b7969a5b1d0661a072a5e1d06d78e54ff39b989206cfe9ec WatchSource:0}: Error finding container 26ef0fa9eb4f3943b7969a5b1d0661a072a5e1d06d78e54ff39b989206cfe9ec: Status 404 returned error can't find the container with id 26ef0fa9eb4f3943b7969a5b1d0661a072a5e1d06d78e54ff39b989206cfe9ec Oct 30 23:43:27.095563 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:27.095538 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr"] Oct 30 23:43:27.438317 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:27.438266 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq"] Oct 30 23:43:27.438562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:27.438522 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:27.438709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:27.438692 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:27.438840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:27.438824 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:27.625314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:27.625278 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" event={"ID":"6474f60e-9d99-45e5-830d-4a668dbd41bb","Type":"ContainerStarted","Data":"d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c"} Oct 30 23:43:27.625582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:27.625557 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" event={"ID":"6474f60e-9d99-45e5-830d-4a668dbd41bb","Type":"ContainerStarted","Data":"c374cc462a0c5664b46b0b45969152f114104a7f39e5752bc3cf052e34786371"} Oct 30 23:43:27.627410 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:27.627369 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" event={"ID":"17a6cbd6-0d33-4498-b390-ee202481a578","Type":"ContainerStarted","Data":"26ef0fa9eb4f3943b7969a5b1d0661a072a5e1d06d78e54ff39b989206cfe9ec"} Oct 30 23:43:27.668970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:27.668838 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" podStartSLOduration=1.6688172749999999 podStartE2EDuration="1.668817275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:27.666314586 +0000 UTC m=+1419.185315677" watchObservedRunningTime="2026-10-30 23:43:27.668817275 +0000 UTC m=+1419.187818367" Oct 30 23:43:28.039101 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:28.039014 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:28.040029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:28.039761 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 23:43:28.040797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:28.040751 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" podUID="6474f60e-9d99-45e5-830d-4a668dbd41bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 23:43:28.638129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:28.638094 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" event={"ID":"17a6cbd6-0d33-4498-b390-ee202481a578","Type":"ContainerStarted","Data":"de3b440649266a60828e1cc204b3aca921ff7157f4574d56ec321ccfe87b3c9a"} Oct 30 23:43:28.638303 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:28.638152 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:28.667266 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:28.667214 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" podStartSLOduration=2.073712047 podStartE2EDuration="2.667199031s" totalImagesPullingTime="593.486984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:28.660728297 +0000 UTC m=+1420.179729386" watchObservedRunningTime="2026-10-30 23:43:28.667199031 +0000 UTC m=+1420.186200120" Oct 30 23:43:29.037088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:29.036799 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 23:43:29.037088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:29.036868 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" podUID="6474f60e-9d99-45e5-830d-4a668dbd41bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 23:43:29.434837 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:43:29.434803 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:43:30.036215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:30.036177 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 23:43:30.036397 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:30.036239 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" podUID="6474f60e-9d99-45e5-830d-4a668dbd41bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 23:43:31.036628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:31.036587 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 23:43:31.037027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:31.036659 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" podUID="6474f60e-9d99-45e5-830d-4a668dbd41bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 23:43:32.036298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:32.036257 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 23:43:32.036550 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:32.036321 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" podUID="6474f60e-9d99-45e5-830d-4a668dbd41bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 23:43:33.036497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:33.036445 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 23:43:33.037099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:33.036514 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" podUID="6474f60e-9d99-45e5-830d-4a668dbd41bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 23:43:34.036561 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.036518 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 23:43:34.036997 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.036604 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" podUID="6474f60e-9d99-45e5-830d-4a668dbd41bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 23:43:34.167395 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.167358 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs"] Oct 30 23:43:34.172906 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.172880 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.186623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.186587 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 23:43:34.186623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.186617 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 23:43:34.186855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.186706 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qbh58\"" Oct 30 23:43:34.186855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.186714 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 23:43:34.201743 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.201712 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs"] Oct 30 23:43:34.279828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.279787 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.280104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.280084 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.280372 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.280353 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.280587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.280568 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhsj4\" (UniqueName: \"kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-kube-api-access-fhsj4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.280748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.280730 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.281099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.281074 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.287834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.287752 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/363febe9-9102-48c9-8faa-09bf49151bf0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.395722 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.395681 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/363febe9-9102-48c9-8faa-09bf49151bf0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.395943 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.395796 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.395943 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.395838 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.396069 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.396000 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.396128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.396104 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhsj4\" (UniqueName: \"kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-kube-api-access-fhsj4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.396181 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.396139 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.396290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.396274 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.396735 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.396707 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.398305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.398281 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/363febe9-9102-48c9-8faa-09bf49151bf0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.398605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.398583 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.398952 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.398925 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.399062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.399047 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.408641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.408616 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhsj4\" (UniqueName: \"kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-kube-api-access-fhsj4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.409141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.409123 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mxggs\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.488763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.488731 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:34.654262 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.654232 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:43:34.816626 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:34.816449 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs"] Oct 30 23:43:35.404832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:35.404796 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:35.405446 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:35.405420 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:35.406441 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:35.406416 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:43:35.709114 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:35.709018 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" event={"ID":"363febe9-9102-48c9-8faa-09bf49151bf0","Type":"ContainerStarted","Data":"6fce487599129fc1d9adaa23349421c3b96107e00457c968c9b1e584128436ea"} Oct 30 23:43:37.631345 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:37.631315 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" Oct 30 23:43:37.948394 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:37.948358 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:37.948547 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:37.948435 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:38.791295 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:38.791260 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" event={"ID":"363febe9-9102-48c9-8faa-09bf49151bf0","Type":"ContainerStarted","Data":"5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da"} Oct 30 23:43:38.793229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:38.793201 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:38.851489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:38.850380 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" podStartSLOduration=1.7250402550000001 podStartE2EDuration="4.850356577s" totalImagesPullingTime="3.125316322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:38.84291995 +0000 UTC m=+1430.361921048" watchObservedRunningTime="2026-10-30 23:43:38.850356577 +0000 UTC m=+1430.369357671" Oct 30 23:43:39.803680 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:39.803650 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:43:48.982488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:48.958384 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:49.227495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.227369 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:49.230866 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.230829 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:49.343748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.343713 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:43:49.429799 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.428691 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:49.465207 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.465163 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:49.552660 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.552611 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:49.584047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.577189 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:49.622689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.622575 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:49.632487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.630016 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj"] Oct 30 23:43:49.714488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.691322 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:49.736608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.736557 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:43:49.749718 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.740172 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:43:49.749718 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.740245 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:43:49.795633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.795589 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:49.873552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.820963 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj\" (UID: \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:49.874044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.874012 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq29b\" (UniqueName: \"kubernetes.io/projected/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-kube-api-access-cq29b\") pod \"maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj\" (UID: \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:49.874245 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.874224 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj\" (UID: \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:49.874789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.873817 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:49.960598 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.960546 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:49.962676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.962641 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:43:49.971389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.971361 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:43:49.984499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.979136 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj\" (UID: \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:49.984499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.980450 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq29b\" (UniqueName: \"kubernetes.io/projected/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-kube-api-access-cq29b\") pod \"maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj\" (UID: \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:49.984499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.980507 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj\" (UID: \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:49.986508 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:49.979942 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj\" (UID: \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:50.015287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.015253 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj"] Oct 30 23:43:50.015707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.015684 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:43:50.015919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.015895 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:50.034274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.034207 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq29b\" (UniqueName: \"kubernetes.io/projected/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-kube-api-access-cq29b\") pod \"maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj\" (UID: \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:50.047570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.047539 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj\" (UID: \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:50.138438 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.138337 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:50.245136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.242578 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:43:50.245136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.242946 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded" Oct 30 23:43:50.245136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.243214 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:43:50.290497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.289659 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:50.290497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.290235 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:43:50.345628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.345578 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:50.547915 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.456637 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:50.625672 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.548083 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:43:50.625672 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.548147 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:43:50.625672 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.473819 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:50.625672 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.473852 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:43:50.625672 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.548900 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:43:50.625672 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.578620 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:43:50.625672 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.579155 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:43:50.647373 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.627585 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:43:50.647373 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.628134 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:43:50.702617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.702577 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:43:50.702915 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.702881 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:50.967427 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.967382 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:43:50.970221 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:50.970182 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:51.339654 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:51.339600 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:51.566516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:51.564271 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:51.735340 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:51.666589 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:51.766111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:51.766067 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:43:51.766766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:51.688591 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:43:51.864491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:51.666634 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:51.935805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:51.935752 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:51.936227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:51.936200 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:43:52.021711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.021607 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:52.135533 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.037024 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:43:52.160980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.037151 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:43:52.162894 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:43:52.044504 7570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.416s" Oct 30 23:43:52.163237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.163208 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:43:52.741500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.741404 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:52.751183 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.751133 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:43:52.753408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.753366 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:43:52.753866 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.753828 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:43:52.866637 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.866586 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:52.866845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.866763 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:52.867562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.867272 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:43:52.877760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.877701 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:52.949968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:52.949923 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:53.009787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.009670 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:53.128320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.128273 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:53.155616 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.155558 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:53.170330 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.170285 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:53.424659 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.424612 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:53.737549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.737308 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:53.852495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.846307 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:43:53.887564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.847201 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:43:54.059574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.847235 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:43:54.059574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.847256 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 23:43:54.059574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.847285 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:54.109521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.874525 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:54.122817 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.874562 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:43:54.219387 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:43:54.219352 7570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.465s" Oct 30 23:43:54.219743 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:54.219723 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:43:54.219899 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:54.219886 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:43:54.220036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:54.220024 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:43:54.261532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:53.874583 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:43:54.326552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:54.148600 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:54.382623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:54.319342 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:54.405539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:54.192217 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:54.444514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:54.029564 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:54.564487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:54.539417 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:55.054488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.052550 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 30 23:43:55.054488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.053373 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:43:55.235254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.235209 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:43:55.274363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.274310 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:55.305398 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.305286 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" event={"ID":"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15","Type":"ContainerStarted","Data":"b3b7589db471348e43233d3b054936d3ee044145fb7bebba5bece72bbe3012b0"} Oct 30 23:43:55.326485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.322496 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:43:55.336492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.330649 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:43:55.369765 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.337646 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:43:55.369765 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.364775 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:43:55.407292 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.407236 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:55.460849 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.460807 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:55.464719 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.464678 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:43:55.465259 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.465225 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:43:55.467447 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.467420 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj"] Oct 30 23:43:55.571657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.570547 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:55.571657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.570549 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:55.691810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.691775 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:43:55.972487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:55.972357 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:43:56.089533 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.088486 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:56.348963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.348927 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:56.353486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.349885 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 30 23:43:56.426611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.389155 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:56.426611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.389540 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:56.426611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.389808 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:56.426611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.411548 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:56.467092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.467048 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:56.470486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.469677 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:56.470486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.470005 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:43:56.580069 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.578339 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:43:56.633118 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.633038 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:43:56.758107 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.758067 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:43:56.770136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.770024 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:43:56.770429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.770413 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:43:56.794488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.793695 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:43:56.875536 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.875508 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:43:56.882986 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.882961 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:43:56.883264 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.883220 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:43:56.946119 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.940202 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:43:56.959954 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:56.955981 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:43:57.645951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:57.645867 7570 generic.go:356] "Generic (PLEG): container finished" podID="c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15" containerID="de1c20c3bbac6eaa4680dd5690e282a6612049cdb4837d9b4414afd130212c06" exitCode=0 Oct 30 23:43:57.646334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:57.645950 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" event={"ID":"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15","Type":"ContainerDied","Data":"de1c20c3bbac6eaa4680dd5690e282a6612049cdb4837d9b4414afd130212c06"} Oct 30 23:43:59.020945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.020871 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:59.082663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.082629 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-kube-api-access-cq29b\" (UniqueName: \"kubernetes.io/projected/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-kube-api-access-cq29b\") pod \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\" (UID: \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\") " Oct 30 23:43:59.082867 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.082701 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-xks-ca\") pod \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\" (UID: \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\") " Oct 30 23:43:59.082973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.082953 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-openshift-ca\") pod \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\" (UID: \"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15\") " Oct 30 23:43:59.083577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.083544 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-openshift-ca" pod "c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15" (UID: "c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:43:59.084138 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.084113 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:59.088364 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.088328 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-kube-api-access-cq29b" pod "c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15" (UID: "c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15"). InnerVolumeSpecName "kube-api-access-cq29b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:43:59.108156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.108117 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-xks-ca" pod "c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15" (UID: "c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:43:59.186572 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.186535 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq29b\" (UniqueName: \"kubernetes.io/projected/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-kube-api-access-cq29b\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:59.186572 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.186575 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:43:59.481524 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:43:59.481303 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:43:59.686227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.686195 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" Oct 30 23:43:59.686388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.686236 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj" event={"ID":"c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15","Type":"ContainerDied","Data":"b3b7589db471348e43233d3b054936d3ee044145fb7bebba5bece72bbe3012b0"} Oct 30 23:43:59.686388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.686296 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b7589db471348e43233d3b054936d3ee044145fb7bebba5bece72bbe3012b0" Oct 30 23:43:59.837938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.837904 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9"] Oct 30 23:43:59.838305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:43:59.838272 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" podUID="33a5dec2-0378-450a-9942-4d824434ba36" containerName="istio-proxy" containerID="cri-o://fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147" gracePeriod=30 Oct 30 23:44:00.749513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.749454 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd"] Oct 30 23:44:00.750827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.750807 7570 cpu_manager.go:414] "containerMap: removing container" podUID="c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15" containerName="revoke-keys" Oct 30 23:44:00.750923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.750832 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15" containerName="revoke-keys" Oct 30 23:44:00.751027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.751014 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15" containerName="revoke-keys" Oct 30 23:44:00.756767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.756742 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:00.777060 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.774803 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:44:00.777060 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.774915 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:44:00.787148 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.786531 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd"] Oct 30 23:44:00.871416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.870928 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f78c5\" (UniqueName: \"kubernetes.io/projected/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-kube-api-access-f78c5\") pod \"maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd\" (UID: \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:00.871416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.870981 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd\" (UID: \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:00.871973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.871785 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd\" (UID: \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:00.973713 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.973670 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd\" (UID: \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:00.974034 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.973998 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f78c5\" (UniqueName: \"kubernetes.io/projected/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-kube-api-access-f78c5\") pod \"maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd\" (UID: \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:00.975639 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.975586 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd\" (UID: \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:00.975973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.975949 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd\" (UID: \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:00.979576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.979523 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd\" (UID: \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:00.991364 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:00.990974 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f78c5\" (UniqueName: \"kubernetes.io/projected/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-kube-api-access-f78c5\") pod \"maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd\" (UID: \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:01.093535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:01.093501 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:01.457379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:01.457350 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd"] Oct 30 23:44:01.764033 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:01.762598 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" event={"ID":"474c5e3e-bb01-44f6-ad35-00910d4eb2bd","Type":"ContainerStarted","Data":"f581c12e6023e22ec2d62ba2d95a009abe42b13f9fe4fd927ab8a001fafb63bc"} Oct 30 23:44:02.776673 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:02.776633 7570 generic.go:356] "Generic (PLEG): container finished" podID="474c5e3e-bb01-44f6-ad35-00910d4eb2bd" containerID="e5e05ed1a9f747e172622be8a80a5fbb22b200bab07b6c10ee18d3173f4243b7" exitCode=0 Oct 30 23:44:02.777203 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:02.776712 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" event={"ID":"474c5e3e-bb01-44f6-ad35-00910d4eb2bd","Type":"ContainerDied","Data":"e5e05ed1a9f747e172622be8a80a5fbb22b200bab07b6c10ee18d3173f4243b7"} Oct 30 23:44:04.245864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.245821 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:04.381781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.381677 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-openshift-ca\") pod \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\" (UID: \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\") " Oct 30 23:44:04.381975 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.381874 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-kube-api-access-f78c5\" (UniqueName: \"kubernetes.io/projected/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-kube-api-access-f78c5\") pod \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\" (UID: \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\") " Oct 30 23:44:04.381975 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.381950 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-xks-ca\") pod \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\" (UID: \"474c5e3e-bb01-44f6-ad35-00910d4eb2bd\") " Oct 30 23:44:04.382112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.382052 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-openshift-ca" pod "474c5e3e-bb01-44f6-ad35-00910d4eb2bd" (UID: "474c5e3e-bb01-44f6-ad35-00910d4eb2bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:04.382712 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.382686 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:04.384212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.384186 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-kube-api-access-f78c5" pod "474c5e3e-bb01-44f6-ad35-00910d4eb2bd" (UID: "474c5e3e-bb01-44f6-ad35-00910d4eb2bd"). InnerVolumeSpecName "kube-api-access-f78c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:04.384843 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.384822 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-xks-ca" pod "474c5e3e-bb01-44f6-ad35-00910d4eb2bd" (UID: "474c5e3e-bb01-44f6-ad35-00910d4eb2bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:04.484520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.484478 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f78c5\" (UniqueName: \"kubernetes.io/projected/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-kube-api-access-f78c5\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:04.484520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.484516 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474c5e3e-bb01-44f6-ad35-00910d4eb2bd-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:04.815587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.815499 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" Oct 30 23:44:04.815587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.815524 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd" event={"ID":"474c5e3e-bb01-44f6-ad35-00910d4eb2bd","Type":"ContainerDied","Data":"f581c12e6023e22ec2d62ba2d95a009abe42b13f9fe4fd927ab8a001fafb63bc"} Oct 30 23:44:04.815587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:04.815576 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f581c12e6023e22ec2d62ba2d95a009abe42b13f9fe4fd927ab8a001fafb63bc" Oct 30 23:44:06.928994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:06.915658 7570 patch_prober.go:29] interesting pod/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": read tcp 10.134.0.2:39538->10.134.0.109:15021: read: connection reset by peer" start-of-body= Oct 30 23:44:06.928994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:06.915734 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": read tcp 10.134.0.2:39538->10.134.0.109:15021: read: connection reset by peer" Oct 30 23:44:07.853330 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:07.853294 7570 generic.go:356] "Generic (PLEG): container finished" podID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerID="8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e" exitCode=0 Oct 30 23:44:07.853597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:07.853377 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" event={"ID":"a7d7499f-a997-4d6e-9c62-57ce12871c43","Type":"ContainerDied","Data":"8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e"} Oct 30 23:44:07.854008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:07.853988 7570 scope.go:122] "RemoveContainer" containerID="8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e" Oct 30 23:44:07.855932 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:07.855901 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:44:07.856055 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:07.855978 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:44:07.856055 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:07.856016 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:44:08.621491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.614068 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.689803 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-socket\") pod \"33a5dec2-0378-450a-9942-4d824434ba36\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.689857 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a5dec2-0378-450a-9942-4d824434ba36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a5dec2-0378-450a-9942-4d824434ba36-istiod-ca-cert\") pod \"33a5dec2-0378-450a-9942-4d824434ba36\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.689985 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-credential-socket\") pod \"33a5dec2-0378-450a-9942-4d824434ba36\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.690065 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-data\") pod \"33a5dec2-0378-450a-9942-4d824434ba36\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.690174 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-certs\") pod \"33a5dec2-0378-450a-9942-4d824434ba36\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.690207 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-istio-token\" (UniqueName: \"kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-istio-token\") pod \"33a5dec2-0378-450a-9942-4d824434ba36\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.690274 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-kube-api-access-g7xbw\" (UniqueName: \"kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-kube-api-access-g7xbw\") pod \"33a5dec2-0378-450a-9942-4d824434ba36\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.690377 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33a5dec2-0378-450a-9942-4d824434ba36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a5dec2-0378-450a-9942-4d824434ba36-istio-podinfo\") pod \"33a5dec2-0378-450a-9942-4d824434ba36\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.690543 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-envoy\") pod \"33a5dec2-0378-450a-9942-4d824434ba36\" (UID: \"33a5dec2-0378-450a-9942-4d824434ba36\") " Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.692013 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-certs" pod "33a5dec2-0378-450a-9942-4d824434ba36" (UID: "33a5dec2-0378-450a-9942-4d824434ba36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.692542 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-socket" pod "33a5dec2-0378-450a-9942-4d824434ba36" (UID: "33a5dec2-0378-450a-9942-4d824434ba36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.693653 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-credential-socket" pod "33a5dec2-0378-450a-9942-4d824434ba36" (UID: "33a5dec2-0378-450a-9942-4d824434ba36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.693798 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a5dec2-0378-450a-9942-4d824434ba36-istiod-ca-cert" pod "33a5dec2-0378-450a-9942-4d824434ba36" (UID: "33a5dec2-0378-450a-9942-4d824434ba36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.693851 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-data" pod "33a5dec2-0378-450a-9942-4d824434ba36" (UID: "33a5dec2-0378-450a-9942-4d824434ba36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.694098 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.694117 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.694130 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.694142 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:08.708491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.694155 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a5dec2-0378-450a-9942-4d824434ba36-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:08.716555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.716509 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33a5dec2-0378-450a-9942-4d824434ba36-istio-podinfo" pod "33a5dec2-0378-450a-9942-4d824434ba36" (UID: "33a5dec2-0378-450a-9942-4d824434ba36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:44:08.716856 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.716826 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-kube-api-access-g7xbw" pod "33a5dec2-0378-450a-9942-4d824434ba36" (UID: "33a5dec2-0378-450a-9942-4d824434ba36"). InnerVolumeSpecName "kube-api-access-g7xbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:08.716965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.716878 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-envoy" pod "33a5dec2-0378-450a-9942-4d824434ba36" (UID: "33a5dec2-0378-450a-9942-4d824434ba36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:08.717589 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.717568 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-istio-token" pod "33a5dec2-0378-450a-9942-4d824434ba36" (UID: "33a5dec2-0378-450a-9942-4d824434ba36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:08.811838 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.811794 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a5dec2-0378-450a-9942-4d824434ba36-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:08.811838 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.811827 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:08.811838 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.811841 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7xbw\" (UniqueName: \"kubernetes.io/projected/33a5dec2-0378-450a-9942-4d824434ba36-kube-api-access-g7xbw\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:08.812085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.811856 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a5dec2-0378-450a-9942-4d824434ba36-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:08.863845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.863803 7570 generic.go:356] "Generic (PLEG): container finished" podID="33a5dec2-0378-450a-9942-4d824434ba36" containerID="fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147" exitCode=0 Oct 30 23:44:08.864046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.863857 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" Oct 30 23:44:08.864046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.863892 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" event={"ID":"33a5dec2-0378-450a-9942-4d824434ba36","Type":"ContainerDied","Data":"fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147"} Oct 30 23:44:08.864046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.863964 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9" event={"ID":"33a5dec2-0378-450a-9942-4d824434ba36","Type":"ContainerDied","Data":"fb11cf87fbeac137d1d92b49d7d1a7b0831157a88801fdefbbb732dd4bba9c72"} Oct 30 23:44:08.864046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.863988 7570 scope.go:122] "RemoveContainer" containerID="fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147" Oct 30 23:44:08.866607 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.866575 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" event={"ID":"a7d7499f-a997-4d6e-9c62-57ce12871c43","Type":"ContainerStarted","Data":"80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c"} Oct 30 23:44:08.879418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.879378 7570 scope.go:122] "RemoveContainer" containerID="fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147" Oct 30 23:44:08.879754 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:44:08.879727 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147\": container with ID starting with fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147 not found: ID does not exist" containerID="fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147" Oct 30 23:44:08.879860 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.879765 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147"} err="failed to get container status \"fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147\": rpc error: code = NotFound desc = could not find container \"fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147\": container with ID starting with fcb7b66868834b07084bad0e413f755a76f591d5d7080bd1c9e1ccc8d4e02147 not found: ID does not exist" Oct 30 23:44:08.940392 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.940351 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9"] Oct 30 23:44:08.945250 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:08.945226 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ebbe654c-openshift-default-5bf4c69d5c-pnxw9"] Oct 30 23:44:09.054708 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:09.054673 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7"] Oct 30 23:44:09.277840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:09.277756 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:44:09.277999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:09.277948 7570 patch_prober.go:29] interesting pod/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 23:44:09.278041 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:09.277995 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 23:44:10.179616 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.179585 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz"] Oct 30 23:44:10.180350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.180335 7570 cpu_manager.go:414] "containerMap: removing container" podUID="33a5dec2-0378-450a-9942-4d824434ba36" containerName="istio-proxy" Oct 30 23:44:10.180395 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.180351 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a5dec2-0378-450a-9942-4d824434ba36" containerName="istio-proxy" Oct 30 23:44:10.180395 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.180380 7570 cpu_manager.go:414] "containerMap: removing container" podUID="474c5e3e-bb01-44f6-ad35-00910d4eb2bd" containerName="revoke-keys" Oct 30 23:44:10.180395 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.180386 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474c5e3e-bb01-44f6-ad35-00910d4eb2bd" containerName="revoke-keys" Oct 30 23:44:10.180537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.180529 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="474c5e3e-bb01-44f6-ad35-00910d4eb2bd" containerName="revoke-keys" Oct 30 23:44:10.180572 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.180551 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a5dec2-0378-450a-9942-4d824434ba36" containerName="istio-proxy" Oct 30 23:44:10.185064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.185045 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.226437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.225629 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz"] Oct 30 23:44:10.264194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.264148 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.264393 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.264229 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.264393 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.264374 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.264544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.264413 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.264610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.264558 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.264667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.264609 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8k87\" (UniqueName: \"kubernetes.io/projected/9a41f90f-265b-4a47-a2bc-9239bbd1822b-kube-api-access-c8k87\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.264729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.264700 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.264784 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.264767 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.264915 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.264896 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.278004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.277975 7570 patch_prober.go:29] interesting pod/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 23:44:10.278121 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.278032 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 23:44:10.366845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.366810 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.366884 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.366905 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8k87\" (UniqueName: \"kubernetes.io/projected/9a41f90f-265b-4a47-a2bc-9239bbd1822b-kube-api-access-c8k87\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.367065 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.367176 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.367207 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.367292 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.367388 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367506 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.367390 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.367499 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.367528 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367680 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.367602 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367680 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.367626 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.367680 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.367669 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.369947 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.369916 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.370066 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.370033 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.381163 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.381134 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a41f90f-265b-4a47-a2bc-9239bbd1822b-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.381387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.381365 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8k87\" (UniqueName: \"kubernetes.io/projected/9a41f90f-265b-4a47-a2bc-9239bbd1822b-kube-api-access-c8k87\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p5kqz\" (UID: \"9a41f90f-265b-4a47-a2bc-9239bbd1822b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.537110 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.537020 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:10.626408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.626372 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a5dec2-0378-450a-9942-4d824434ba36" path="/var/lib/kubelet/pods/33a5dec2-0378-450a-9942-4d824434ba36/volumes" Oct 30 23:44:10.873245 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.873208 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz"] Oct 30 23:44:10.877655 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:44:10.877624 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a41f90f_265b_4a47_a2bc_9239bbd1822b.slice/crio-10019eda0a87388a154475c867ff305924b3d3d443f1ac02f99a9c88861a388c WatchSource:0}: Error finding container 10019eda0a87388a154475c867ff305924b3d3d443f1ac02f99a9c88861a388c: Status 404 returned error can't find the container with id 10019eda0a87388a154475c867ff305924b3d3d443f1ac02f99a9c88861a388c Oct 30 23:44:10.881109 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.881064 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:44:10.881215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.881177 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:44:10.881263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:10.881221 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:44:11.060122 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:11.060086 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc"] Oct 30 23:44:11.277625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:11.277537 7570 patch_prober.go:29] interesting pod/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 23:44:11.277625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:11.277604 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 23:44:11.944006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:11.943968 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" event={"ID":"9a41f90f-265b-4a47-a2bc-9239bbd1822b","Type":"ContainerStarted","Data":"a3c552428ece605aa5fb744fef877cca1a6cda6f89a1f4dbf9c2e1982623c6f9"} Oct 30 23:44:11.944179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:11.944023 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" event={"ID":"9a41f90f-265b-4a47-a2bc-9239bbd1822b","Type":"ContainerStarted","Data":"10019eda0a87388a154475c867ff305924b3d3d443f1ac02f99a9c88861a388c"} Oct 30 23:44:11.975608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:11.975547 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podStartSLOduration=1.975534328 podStartE2EDuration="1.975534328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:11.972078252 +0000 UTC m=+1463.491079343" watchObservedRunningTime="2026-10-30 23:44:11.975534328 +0000 UTC m=+1463.494535417" Oct 30 23:44:12.278185 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:12.278092 7570 patch_prober.go:29] interesting pod/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 23:44:12.278185 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:12.278161 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 23:44:12.545568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:12.545426 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:12.548943 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:12.548902 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 23:44:12.549110 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:12.548969 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 23:44:13.050493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:13.046996 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr"] Oct 30 23:44:13.062488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:13.061811 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" podUID="17a6cbd6-0d33-4498-b390-ee202481a578" containerName="maas-api" containerID="cri-o://de3b440649266a60828e1cc204b3aca921ff7157f4574d56ec321ccfe87b3c9a" gracePeriod=30 Oct 30 23:44:13.278486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:13.277883 7570 patch_prober.go:29] interesting pod/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 23:44:13.278486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:13.277945 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 23:44:13.553495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:13.551596 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 23:44:13.553495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:13.551675 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 23:44:14.058237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.058106 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" podUID="e96b553f-6765-4476-a306-4c5702af33c5" containerName="payload-processing" containerID="cri-o://d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785" gracePeriod=25 Oct 30 23:44:14.231321 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.229923 7570 generic.go:356] "Generic (PLEG): container finished" podID="17a6cbd6-0d33-4498-b390-ee202481a578" containerID="de3b440649266a60828e1cc204b3aca921ff7157f4574d56ec321ccfe87b3c9a" exitCode=0 Oct 30 23:44:14.231321 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.231213 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj"] Oct 30 23:44:14.231321 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.231249 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" event={"ID":"17a6cbd6-0d33-4498-b390-ee202481a578","Type":"ContainerDied","Data":"de3b440649266a60828e1cc204b3aca921ff7157f4574d56ec321ccfe87b3c9a"} Oct 30 23:44:14.240432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.237069 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d34ab3fd-xkcwj"] Oct 30 23:44:14.245715 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.245689 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:44:14.283018 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.281519 7570 patch_prober.go:29] interesting pod/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 23:44:14.283018 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.281596 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 23:44:14.434513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.431604 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-metrics-certs\") pod \"17a6cbd6-0d33-4498-b390-ee202481a578\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " Oct 30 23:44:14.434513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.431791 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a6cbd6-0d33-4498-b390-ee202481a578-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a6cbd6-0d33-4498-b390-ee202481a578-openshift-service-ca\") pod \"17a6cbd6-0d33-4498-b390-ee202481a578\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " Oct 30 23:44:14.434513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.431864 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a6cbd6-0d33-4498-b390-ee202481a578-kube-api-access-jc9jw\" (UniqueName: \"kubernetes.io/projected/17a6cbd6-0d33-4498-b390-ee202481a578-kube-api-access-jc9jw\") pod \"17a6cbd6-0d33-4498-b390-ee202481a578\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " Oct 30 23:44:14.434513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.431924 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-maas-api-tls\") pod \"17a6cbd6-0d33-4498-b390-ee202481a578\" (UID: \"17a6cbd6-0d33-4498-b390-ee202481a578\") " Oct 30 23:44:14.436977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.436932 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a6cbd6-0d33-4498-b390-ee202481a578-openshift-service-ca" pod "17a6cbd6-0d33-4498-b390-ee202481a578" (UID: "17a6cbd6-0d33-4498-b390-ee202481a578"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:14.467999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.451939 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-maas-api-tls" pod "17a6cbd6-0d33-4498-b390-ee202481a578" (UID: "17a6cbd6-0d33-4498-b390-ee202481a578"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:14.471574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.471501 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a6cbd6-0d33-4498-b390-ee202481a578-kube-api-access-jc9jw" pod "17a6cbd6-0d33-4498-b390-ee202481a578" (UID: "17a6cbd6-0d33-4498-b390-ee202481a578"). InnerVolumeSpecName "kube-api-access-jc9jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:14.513487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.512575 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-metrics-certs" pod "17a6cbd6-0d33-4498-b390-ee202481a578" (UID: "17a6cbd6-0d33-4498-b390-ee202481a578"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:14.552853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.548055 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:14.552853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.548091 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a6cbd6-0d33-4498-b390-ee202481a578-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:14.552853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.548105 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc9jw\" (UniqueName: \"kubernetes.io/projected/17a6cbd6-0d33-4498-b390-ee202481a578-kube-api-access-jc9jw\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:14.552853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.548117 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a6cbd6-0d33-4498-b390-ee202481a578-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:14.552853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.550387 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 23:44:14.552853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.550439 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 23:44:14.672410 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.672310 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15" path="/var/lib/kubelet/pods/c3a96cf6-eb7a-4fd3-9d57-aa77e9e2ea15/volumes" Oct 30 23:44:14.735208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.735186 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" Oct 30 23:44:14.769389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.769035 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96b553f-6765-4476-a306-4c5702af33c5-kube-api-access-mvs6k\" (UniqueName: \"kubernetes.io/projected/e96b553f-6765-4476-a306-4c5702af33c5-kube-api-access-mvs6k\") pod \"e96b553f-6765-4476-a306-4c5702af33c5\" (UID: \"e96b553f-6765-4476-a306-4c5702af33c5\") " Oct 30 23:44:14.769659 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.769543 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e96b553f-6765-4476-a306-4c5702af33c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96b553f-6765-4476-a306-4c5702af33c5-plugins-config-volume\") pod \"e96b553f-6765-4476-a306-4c5702af33c5\" (UID: \"e96b553f-6765-4476-a306-4c5702af33c5\") " Oct 30 23:44:14.774402 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.770810 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96b553f-6765-4476-a306-4c5702af33c5-plugins-config-volume" pod "e96b553f-6765-4476-a306-4c5702af33c5" (UID: "e96b553f-6765-4476-a306-4c5702af33c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:14.853498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.848935 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96b553f-6765-4476-a306-4c5702af33c5-kube-api-access-mvs6k" pod "e96b553f-6765-4476-a306-4c5702af33c5" (UID: "e96b553f-6765-4476-a306-4c5702af33c5"). InnerVolumeSpecName "kube-api-access-mvs6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:14.878520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.874823 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvs6k\" (UniqueName: \"kubernetes.io/projected/e96b553f-6765-4476-a306-4c5702af33c5-kube-api-access-mvs6k\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:14.878520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:14.874857 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96b553f-6765-4476-a306-4c5702af33c5-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:15.247527 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.247489 7570 generic.go:356] "Generic (PLEG): container finished" podID="e96b553f-6765-4476-a306-4c5702af33c5" containerID="d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785" exitCode=0 Oct 30 23:44:15.247729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.247616 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" Oct 30 23:44:15.247729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.247627 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" event={"ID":"e96b553f-6765-4476-a306-4c5702af33c5","Type":"ContainerDied","Data":"d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785"} Oct 30 23:44:15.247852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.247727 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7" event={"ID":"e96b553f-6765-4476-a306-4c5702af33c5","Type":"ContainerDied","Data":"62b22fbb03471803a6189a22a11b7d8fba587d1c6b90dc965debcdf3f7741263"} Oct 30 23:44:15.247852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.247759 7570 scope.go:122] "RemoveContainer" containerID="d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785" Oct 30 23:44:15.249754 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.249733 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" event={"ID":"17a6cbd6-0d33-4498-b390-ee202481a578","Type":"ContainerDied","Data":"26ef0fa9eb4f3943b7969a5b1d0661a072a5e1d06d78e54ff39b989206cfe9ec"} Oct 30 23:44:15.249902 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.249796 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr" Oct 30 23:44:15.264477 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.264435 7570 scope.go:122] "RemoveContainer" containerID="d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785" Oct 30 23:44:15.264823 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:44:15.264801 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785\": container with ID starting with d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785 not found: ID does not exist" containerID="d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785" Oct 30 23:44:15.264882 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.264835 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785"} err="failed to get container status \"d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785\": rpc error: code = NotFound desc = could not find container \"d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785\": container with ID starting with d252fa9405922620de13a123dabbaffe8570cbbcb31755e68e0e95d76ef10785 not found: ID does not exist" Oct 30 23:44:15.264882 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.264858 7570 scope.go:122] "RemoveContainer" containerID="de3b440649266a60828e1cc204b3aca921ff7157f4574d56ec321ccfe87b3c9a" Oct 30 23:44:15.283050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.283021 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:44:15.283746 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.283726 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:44:15.317528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.305325 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:44:15.320444 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.320413 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr"] Oct 30 23:44:15.329052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.329022 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d34ab3fd-695b6c7d66-jz7mr"] Oct 30 23:44:15.347609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.347579 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7"] Oct 30 23:44:15.352279 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.352260 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d34ab3fd-66ccf5984f-kxnf7"] Oct 30 23:44:15.520595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.520500 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd"] Oct 30 23:44:15.541835 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.540399 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 23:44:15.541835 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:15.540497 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 23:44:16.061236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.061182 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" podUID="251680aa-f28b-4c53-be4f-47b0a6de4cef" containerName="payload-processing" containerID="cri-o://89a65c061be5cdc9c2845077a37a9630744fab9938adf1cf555c12cf8d721d31" gracePeriod=25 Oct 30 23:44:16.263612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.263580 7570 generic.go:356] "Generic (PLEG): container finished" podID="251680aa-f28b-4c53-be4f-47b0a6de4cef" containerID="89a65c061be5cdc9c2845077a37a9630744fab9938adf1cf555c12cf8d721d31" exitCode=0 Oct 30 23:44:16.263840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.263714 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" event={"ID":"251680aa-f28b-4c53-be4f-47b0a6de4cef","Type":"ContainerDied","Data":"89a65c061be5cdc9c2845077a37a9630744fab9938adf1cf555c12cf8d721d31"} Oct 30 23:44:16.265393 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.265367 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" secret="" err="secret \"e2e-ait-d34ab3fd-openshift-default-dockercfg-ss8cz\" not found" Oct 30 23:44:16.458485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.452549 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" Oct 30 23:44:16.538511 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.538431 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 23:44:16.538699 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.538535 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 23:44:16.555142 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.555110 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251680aa-f28b-4c53-be4f-47b0a6de4cef-kube-api-access-27pqc\" (UniqueName: \"kubernetes.io/projected/251680aa-f28b-4c53-be4f-47b0a6de4cef-kube-api-access-27pqc\") pod \"251680aa-f28b-4c53-be4f-47b0a6de4cef\" (UID: \"251680aa-f28b-4c53-be4f-47b0a6de4cef\") " Oct 30 23:44:16.555454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.555434 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251680aa-f28b-4c53-be4f-47b0a6de4cef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251680aa-f28b-4c53-be4f-47b0a6de4cef-plugins-config-volume\") pod \"251680aa-f28b-4c53-be4f-47b0a6de4cef\" (UID: \"251680aa-f28b-4c53-be4f-47b0a6de4cef\") " Oct 30 23:44:16.555905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.555876 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251680aa-f28b-4c53-be4f-47b0a6de4cef-plugins-config-volume" pod "251680aa-f28b-4c53-be4f-47b0a6de4cef" (UID: "251680aa-f28b-4c53-be4f-47b0a6de4cef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:16.556228 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.556210 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251680aa-f28b-4c53-be4f-47b0a6de4cef-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:16.558065 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.558040 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251680aa-f28b-4c53-be4f-47b0a6de4cef-kube-api-access-27pqc" pod "251680aa-f28b-4c53-be4f-47b0a6de4cef" (UID: "251680aa-f28b-4c53-be4f-47b0a6de4cef"). InnerVolumeSpecName "kube-api-access-27pqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:16.632251 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.632161 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a6cbd6-0d33-4498-b390-ee202481a578" path="/var/lib/kubelet/pods/17a6cbd6-0d33-4498-b390-ee202481a578/volumes" Oct 30 23:44:16.632896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.632868 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e96b553f-6765-4476-a306-4c5702af33c5" path="/var/lib/kubelet/pods/e96b553f-6765-4476-a306-4c5702af33c5/volumes" Oct 30 23:44:16.658417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:16.658393 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27pqc\" (UniqueName: \"kubernetes.io/projected/251680aa-f28b-4c53-be4f-47b0a6de4cef-kube-api-access-27pqc\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:17.159163 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.159127 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p"] Oct 30 23:44:17.159963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.159945 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e96b553f-6765-4476-a306-4c5702af33c5" containerName="payload-processing" Oct 30 23:44:17.159963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.159965 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96b553f-6765-4476-a306-4c5702af33c5" containerName="payload-processing" Oct 30 23:44:17.160102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.159984 7570 cpu_manager.go:414] "containerMap: removing container" podUID="17a6cbd6-0d33-4498-b390-ee202481a578" containerName="maas-api" Oct 30 23:44:17.160102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.159989 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a6cbd6-0d33-4498-b390-ee202481a578" containerName="maas-api" Oct 30 23:44:17.160102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.160030 7570 cpu_manager.go:414] "containerMap: removing container" podUID="251680aa-f28b-4c53-be4f-47b0a6de4cef" containerName="payload-processing" Oct 30 23:44:17.160102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.160035 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251680aa-f28b-4c53-be4f-47b0a6de4cef" containerName="payload-processing" Oct 30 23:44:17.160243 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.160129 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96b553f-6765-4476-a306-4c5702af33c5" containerName="payload-processing" Oct 30 23:44:17.160243 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.160139 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a6cbd6-0d33-4498-b390-ee202481a578" containerName="maas-api" Oct 30 23:44:17.160243 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.160154 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="251680aa-f28b-4c53-be4f-47b0a6de4cef" containerName="payload-processing" Oct 30 23:44:17.165142 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.165123 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.171920 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.171895 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ab09c434-openshift-default-dockercfg-b6zgt\"" Oct 30 23:44:17.223886 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.223850 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p"] Oct 30 23:44:17.280332 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.280293 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-data\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.280829 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.280682 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-certs\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.281065 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.281038 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-podinfo\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.281329 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.281312 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-socket\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.286587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.286556 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-token\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.286754 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.286735 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istiod-ca-cert\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.287211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.287184 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-envoy\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.287443 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.287423 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-credential-socket\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.294495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.287740 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgxr7\" (UniqueName: \"kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-kube-api-access-cgxr7\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.496266 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.496110 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" containerID="cri-o://80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c" gracePeriod=30 Oct 30 23:44:17.497454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.497431 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" Oct 30 23:44:17.498701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.498659 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc" event={"ID":"251680aa-f28b-4c53-be4f-47b0a6de4cef","Type":"ContainerDied","Data":"0ecb8f704d2e57a79c17b92b5ce94375954338ae722944a29fbecce0e7b0318d"} Oct 30 23:44:17.498853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.498769 7570 scope.go:122] "RemoveContainer" containerID="89a65c061be5cdc9c2845077a37a9630744fab9938adf1cf555c12cf8d721d31" Oct 30 23:44:17.546407 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.546368 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 23:44:17.546785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.546751 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 23:44:17.564610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.564571 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-envoy\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.564787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.564694 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-credential-socket\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.567582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.566395 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgxr7\" (UniqueName: \"kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-kube-api-access-cgxr7\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.567582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.566549 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-data\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.567582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.566650 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-certs\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.567582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.566759 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-podinfo\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.567582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.566898 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-socket\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.567582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.566943 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-token\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.567582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.566974 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istiod-ca-cert\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.568123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.567934 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istiod-ca-cert\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.568213 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.568190 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-data\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.568651 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.568633 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-socket\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.569291 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.569272 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-credential-socket\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.569806 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.569607 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-certs\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.571084 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.571061 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-podinfo\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.585111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.585084 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgxr7\" (UniqueName: \"kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-kube-api-access-cgxr7\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.622495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.616085 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-envoy\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.635562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.635483 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-token\") pod \"e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:17.657444 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.657416 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc"] Oct 30 23:44:17.662628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.662535 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dd98b91-7b46b9cb4c-4spzc"] Oct 30 23:44:17.780373 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:17.780283 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:18.090250 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:44:18.090196 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d61ecc_0548_44d5_8852_b00e3b5b19da.slice/crio-24ebbb76c0feb9ff66f9a954167251944c5a15b1857ed623d3918d4b6aa13089 WatchSource:0}: Error finding container 24ebbb76c0feb9ff66f9a954167251944c5a15b1857ed623d3918d4b6aa13089: Status 404 returned error can't find the container with id 24ebbb76c0feb9ff66f9a954167251944c5a15b1857ed623d3918d4b6aa13089 Oct 30 23:44:18.095002 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:18.094972 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p"] Oct 30 23:44:18.095478 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:18.095425 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:44:18.095557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:18.095515 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:44:18.095557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:18.095552 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:44:18.512400 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:18.512293 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" event={"ID":"b0d61ecc-0548-44d5-8852-b00e3b5b19da","Type":"ContainerStarted","Data":"66c2f14fe59c02ee855dcdf9ed62ef6390cd8866673457cbcc77cdc852ad4f25"} Oct 30 23:44:18.512400 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:18.512364 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" event={"ID":"b0d61ecc-0548-44d5-8852-b00e3b5b19da","Type":"ContainerStarted","Data":"24ebbb76c0feb9ff66f9a954167251944c5a15b1857ed623d3918d4b6aa13089"} Oct 30 23:44:18.537647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:18.537601 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 23:44:18.537844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:18.537659 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 23:44:18.549281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:18.549215 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" podStartSLOduration=1.549196817 podStartE2EDuration="1.549196817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:18.544565818 +0000 UTC m=+1470.063566917" watchObservedRunningTime="2026-10-30 23:44:18.549196817 +0000 UTC m=+1470.068197909" Oct 30 23:44:18.606201 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:18.606172 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251680aa-f28b-4c53-be4f-47b0a6de4cef" path="/var/lib/kubelet/pods/251680aa-f28b-4c53-be4f-47b0a6de4cef/volumes" Oct 30 23:44:18.781225 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:18.781132 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:19.786565 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:19.786534 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:20.104048 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:20.104006 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v"] Oct 30 23:44:20.117148 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:20.117099 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca170e97-t728v"] Oct 30 23:44:20.533294 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:20.533205 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:20.534514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:20.534448 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:44:20.535437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:20.535415 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:20.536336 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:20.536315 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:20.537026 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:20.537009 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:44:20.620361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:20.620329 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eba9836-e9a4-4043-a10d-a8a26eaa9ac9" path="/var/lib/kubelet/pods/8eba9836-e9a4-4043-a10d-a8a26eaa9ac9/volumes" Oct 30 23:44:21.850628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:21.850596 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb"] Oct 30 23:44:21.856778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:21.856753 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" Oct 30 23:44:21.867037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:21.867009 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ab09c434\"" Oct 30 23:44:21.874587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:21.874565 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb"] Oct 30 23:44:21.959189 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:21.959143 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6pb\" (UniqueName: \"kubernetes.io/projected/a67c142d-eada-462c-a292-cd01ce48ed0e-kube-api-access-bc6pb\") pod \"payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb\" (UID: \"a67c142d-eada-462c-a292-cd01ce48ed0e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" Oct 30 23:44:21.959551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:21.959532 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67c142d-eada-462c-a292-cd01ce48ed0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb\" (UID: \"a67c142d-eada-462c-a292-cd01ce48ed0e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" Oct 30 23:44:22.068487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.061257 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6pb\" (UniqueName: \"kubernetes.io/projected/a67c142d-eada-462c-a292-cd01ce48ed0e-kube-api-access-bc6pb\") pod \"payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb\" (UID: \"a67c142d-eada-462c-a292-cd01ce48ed0e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" Oct 30 23:44:22.068487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.061656 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67c142d-eada-462c-a292-cd01ce48ed0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb\" (UID: \"a67c142d-eada-462c-a292-cd01ce48ed0e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" Oct 30 23:44:22.088512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.085079 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67c142d-eada-462c-a292-cd01ce48ed0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb\" (UID: \"a67c142d-eada-462c-a292-cd01ce48ed0e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" Oct 30 23:44:22.116537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.115426 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6pb\" (UniqueName: \"kubernetes.io/projected/a67c142d-eada-462c-a292-cd01ce48ed0e-kube-api-access-bc6pb\") pod \"payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb\" (UID: \"a67c142d-eada-462c-a292-cd01ce48ed0e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" Oct 30 23:44:22.176082 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.176035 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" Oct 30 23:44:22.488003 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:44:22.487972 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67c142d_eada_462c_a292_cd01ce48ed0e.slice/crio-6ce5de1e2d915c79740c6e6ae22cb2bb50b9eede49851700f23b8550cda01bae WatchSource:0}: Error finding container 6ce5de1e2d915c79740c6e6ae22cb2bb50b9eede49851700f23b8550cda01bae: Status 404 returned error can't find the container with id 6ce5de1e2d915c79740c6e6ae22cb2bb50b9eede49851700f23b8550cda01bae Oct 30 23:44:22.489222 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.489197 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb"] Oct 30 23:44:22.557085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.557052 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" event={"ID":"a67c142d-eada-462c-a292-cd01ce48ed0e","Type":"ContainerStarted","Data":"6ce5de1e2d915c79740c6e6ae22cb2bb50b9eede49851700f23b8550cda01bae"} Oct 30 23:44:22.852893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.852867 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:44:22.986516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.985560 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-data\") pod \"a7d7499f-a997-4d6e-9c62-57ce12871c43\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " Oct 30 23:44:22.986516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.985711 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-certs\") pod \"a7d7499f-a997-4d6e-9c62-57ce12871c43\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " Oct 30 23:44:22.986516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.985751 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-kube-api-access-572t4\" (UniqueName: \"kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-kube-api-access-572t4\") pod \"a7d7499f-a997-4d6e-9c62-57ce12871c43\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " Oct 30 23:44:22.986516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.985815 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-envoy\") pod \"a7d7499f-a997-4d6e-9c62-57ce12871c43\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " Oct 30 23:44:22.986516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.986484 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-token\" (UniqueName: \"kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-token\") pod \"a7d7499f-a997-4d6e-9c62-57ce12871c43\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " Oct 30 23:44:22.988097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.986568 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-podinfo\") pod \"a7d7499f-a997-4d6e-9c62-57ce12871c43\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " Oct 30 23:44:22.988097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.986693 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d7499f-a997-4d6e-9c62-57ce12871c43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d7499f-a997-4d6e-9c62-57ce12871c43-istiod-ca-cert\") pod \"a7d7499f-a997-4d6e-9c62-57ce12871c43\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " Oct 30 23:44:22.988097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.987730 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d7499f-a997-4d6e-9c62-57ce12871c43-istiod-ca-cert" pod "a7d7499f-a997-4d6e-9c62-57ce12871c43" (UID: "a7d7499f-a997-4d6e-9c62-57ce12871c43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:22.988322 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.988238 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-socket" pod "a7d7499f-a997-4d6e-9c62-57ce12871c43" (UID: "a7d7499f-a997-4d6e-9c62-57ce12871c43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:22.988495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.988452 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-data" pod "a7d7499f-a997-4d6e-9c62-57ce12871c43" (UID: "a7d7499f-a997-4d6e-9c62-57ce12871c43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:22.988648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.988628 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-certs" pod "a7d7499f-a997-4d6e-9c62-57ce12871c43" (UID: "a7d7499f-a997-4d6e-9c62-57ce12871c43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:22.988735 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.986839 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-socket\") pod \"a7d7499f-a997-4d6e-9c62-57ce12871c43\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " Oct 30 23:44:22.988735 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.988719 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-credential-socket\") pod \"a7d7499f-a997-4d6e-9c62-57ce12871c43\" (UID: \"a7d7499f-a997-4d6e-9c62-57ce12871c43\") " Oct 30 23:44:22.989286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.989260 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-credential-socket" pod "a7d7499f-a997-4d6e-9c62-57ce12871c43" (UID: "a7d7499f-a997-4d6e-9c62-57ce12871c43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:22.990075 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.990050 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:22.990183 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.990078 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:22.990183 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.990095 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d7499f-a997-4d6e-9c62-57ce12871c43-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:22.990183 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.990109 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:22.990183 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.990124 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:22.991011 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.990987 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-podinfo" pod "a7d7499f-a997-4d6e-9c62-57ce12871c43" (UID: "a7d7499f-a997-4d6e-9c62-57ce12871c43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:44:22.991927 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.991900 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-envoy" pod "a7d7499f-a997-4d6e-9c62-57ce12871c43" (UID: "a7d7499f-a997-4d6e-9c62-57ce12871c43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:22.992362 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.992342 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-kube-api-access-572t4" pod "a7d7499f-a997-4d6e-9c62-57ce12871c43" (UID: "a7d7499f-a997-4d6e-9c62-57ce12871c43"). InnerVolumeSpecName "kube-api-access-572t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:22.992590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:22.992573 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-token" pod "a7d7499f-a997-4d6e-9c62-57ce12871c43" (UID: "a7d7499f-a997-4d6e-9c62-57ce12871c43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:23.092269 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.092233 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:23.092269 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.092263 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-572t4\" (UniqueName: \"kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-kube-api-access-572t4\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:23.092269 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.092274 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:23.092552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.092282 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d7499f-a997-4d6e-9c62-57ce12871c43-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:23.576341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.576241 7570 generic.go:356] "Generic (PLEG): container finished" podID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerID="80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c" exitCode=0 Oct 30 23:44:23.576341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.576317 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" Oct 30 23:44:23.576341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.576318 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" event={"ID":"a7d7499f-a997-4d6e-9c62-57ce12871c43","Type":"ContainerDied","Data":"80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c"} Oct 30 23:44:23.576689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.576419 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd" event={"ID":"a7d7499f-a997-4d6e-9c62-57ce12871c43","Type":"ContainerDied","Data":"53d7ebb1384f1ee00c64076db01842e78fecfb6a2d9287b7a6677fd30866af19"} Oct 30 23:44:23.576689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.576448 7570 scope.go:122] "RemoveContainer" containerID="80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c" Oct 30 23:44:23.583172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.583128 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" event={"ID":"a67c142d-eada-462c-a292-cd01ce48ed0e","Type":"ContainerStarted","Data":"ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640"} Oct 30 23:44:23.583305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.583267 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" Oct 30 23:44:23.594011 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.593986 7570 scope.go:122] "RemoveContainer" containerID="8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e" Oct 30 23:44:23.606153 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.606129 7570 scope.go:122] "RemoveContainer" containerID="80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c" Oct 30 23:44:23.606489 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:44:23.606450 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c\": container with ID starting with 80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c not found: ID does not exist" containerID="80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c" Oct 30 23:44:23.606573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.606499 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c"} err="failed to get container status \"80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c\": rpc error: code = NotFound desc = could not find container \"80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c\": container with ID starting with 80d0c4bb7341b33da3c86babb830296cd92f62c3aec1c0b57c6aca1a176c246c not found: ID does not exist" Oct 30 23:44:23.606573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.606521 7570 scope.go:122] "RemoveContainer" containerID="8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e" Oct 30 23:44:23.606829 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:44:23.606811 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e\": container with ID starting with 8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e not found: ID does not exist" containerID="8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e" Oct 30 23:44:23.606874 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.606836 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e"} err="failed to get container status \"8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e\": rpc error: code = NotFound desc = could not find container \"8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e\": container with ID starting with 8543bcf2dd88f7e8e0e3caec7719833dcfd4ee7502f96d4852baa87e1ab07c8e not found: ID does not exist" Oct 30 23:44:23.633038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.633006 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd"] Oct 30 23:44:23.644064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.644030 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d34ab3fd-openshift-default-858f78cf99-fk9bd"] Oct 30 23:44:23.665295 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:23.658048 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" podStartSLOduration=2.658029619 podStartE2EDuration="2.658029619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:23.655403625 +0000 UTC m=+1475.174404719" watchObservedRunningTime="2026-10-30 23:44:23.658029619 +0000 UTC m=+1475.177030710" Oct 30 23:44:24.604042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:24.604015 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" path="/var/lib/kubelet/pods/a7d7499f-a997-4d6e-9c62-57ce12871c43/volumes" Oct 30 23:44:26.001908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:26.001876 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd"] Oct 30 23:44:26.008575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:26.008534 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dd98b91-gfrmd"] Oct 30 23:44:26.604336 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:26.604296 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474c5e3e-bb01-44f6-ad35-00910d4eb2bd" path="/var/lib/kubelet/pods/474c5e3e-bb01-44f6-ad35-00910d4eb2bd/volumes" Oct 30 23:44:29.442796 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:44:29.442204 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:44:31.332234 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.332197 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf"] Oct 30 23:44:31.332994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.332976 7570 cpu_manager.go:414] "containerMap: removing container" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" Oct 30 23:44:31.333051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.332996 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" Oct 30 23:44:31.333051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.333036 7570 cpu_manager.go:414] "containerMap: removing container" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" Oct 30 23:44:31.333051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.333041 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" Oct 30 23:44:31.333152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.333144 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" Oct 30 23:44:31.333183 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.333160 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d7499f-a997-4d6e-9c62-57ce12871c43" containerName="istio-proxy" Oct 30 23:44:31.338907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.338878 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:31.345816 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.345787 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:44:31.347145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.347121 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:44:31.351075 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.351055 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf"] Oct 30 23:44:31.362847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.360892 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b487ee95-4jldf\" (UID: \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:31.362847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.362510 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8fdb\" (UniqueName: \"kubernetes.io/projected/231f6ea2-fcf2-416c-adf5-8b195cb0df01-kube-api-access-n8fdb\") pod \"maas-api-revoke-sub-b487ee95-4jldf\" (UID: \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:31.362847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.362580 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b487ee95-4jldf\" (UID: \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:31.463939 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.463904 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b487ee95-4jldf\" (UID: \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:31.464155 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.464016 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8fdb\" (UniqueName: \"kubernetes.io/projected/231f6ea2-fcf2-416c-adf5-8b195cb0df01-kube-api-access-n8fdb\") pod \"maas-api-revoke-sub-b487ee95-4jldf\" (UID: \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:31.464155 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.464054 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b487ee95-4jldf\" (UID: \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:31.464610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.464582 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b487ee95-4jldf\" (UID: \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:31.466874 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.466852 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b487ee95-4jldf\" (UID: \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:31.505452 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.505423 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8fdb\" (UniqueName: \"kubernetes.io/projected/231f6ea2-fcf2-416c-adf5-8b195cb0df01-kube-api-access-n8fdb\") pod \"maas-api-revoke-sub-b487ee95-4jldf\" (UID: \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:31.663579 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:31.663486 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:32.056724 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:44:32.056683 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231f6ea2_fcf2_416c_adf5_8b195cb0df01.slice/crio-df517b7d8186ed3b431fd7846f47e02c84b76c54e7652e69758ac33c636e2916 WatchSource:0}: Error finding container df517b7d8186ed3b431fd7846f47e02c84b76c54e7652e69758ac33c636e2916: Status 404 returned error can't find the container with id df517b7d8186ed3b431fd7846f47e02c84b76c54e7652e69758ac33c636e2916 Oct 30 23:44:32.060515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.059827 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf"] Oct 30 23:44:32.682378 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.682343 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2"] Oct 30 23:44:32.687779 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.687750 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:32.697804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.697620 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ab09c434-serving-cert\"" Oct 30 23:44:32.704695 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.704129 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2"] Oct 30 23:44:32.789456 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.789423 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-maas-api-tls\") pod \"maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:32.789647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.789601 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e10c50-ed06-4152-a626-64a1bc800d53-openshift-service-ca\") pod \"maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:32.789750 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.789722 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-metrics-certs\") pod \"maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:32.789884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.789866 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs4vb\" (UniqueName: \"kubernetes.io/projected/b6e10c50-ed06-4152-a626-64a1bc800d53-kube-api-access-hs4vb\") pod \"maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:32.792108 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.792078 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" event={"ID":"231f6ea2-fcf2-416c-adf5-8b195cb0df01","Type":"ContainerStarted","Data":"d2a4ba1529eb00d22b53bbdbb6fcdf460c9fb8070cc9948d8cfd49316f00de13"} Oct 30 23:44:32.792233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.792158 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" event={"ID":"231f6ea2-fcf2-416c-adf5-8b195cb0df01","Type":"ContainerStarted","Data":"df517b7d8186ed3b431fd7846f47e02c84b76c54e7652e69758ac33c636e2916"} Oct 30 23:44:32.826428 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.824452 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" podStartSLOduration=1.82443364 podStartE2EDuration="1.82443364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:32.812831062 +0000 UTC m=+1484.331832156" watchObservedRunningTime="2026-10-30 23:44:32.82443364 +0000 UTC m=+1484.343434732" Oct 30 23:44:32.898489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.895488 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-metrics-certs\") pod \"maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:32.898489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.895673 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4vb\" (UniqueName: \"kubernetes.io/projected/b6e10c50-ed06-4152-a626-64a1bc800d53-kube-api-access-hs4vb\") pod \"maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:32.898489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.895938 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-maas-api-tls\") pod \"maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:32.898489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.896091 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e10c50-ed06-4152-a626-64a1bc800d53-openshift-service-ca\") pod \"maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:32.898489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.896911 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e10c50-ed06-4152-a626-64a1bc800d53-openshift-service-ca\") pod \"maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:32.900484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.899054 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-metrics-certs\") pod \"maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:32.904757 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.904732 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-maas-api-tls\") pod \"maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:32.914720 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:32.914690 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs4vb\" (UniqueName: \"kubernetes.io/projected/b6e10c50-ed06-4152-a626-64a1bc800d53-kube-api-access-hs4vb\") pod \"maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:33.020430 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:33.020332 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:33.348050 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:44:33.348016 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e10c50_ed06_4152_a626_64a1bc800d53.slice/crio-e66b49def86ef498fe55f2978dda47a22950d20fc62f6206d6a52644cdb59d19 WatchSource:0}: Error finding container e66b49def86ef498fe55f2978dda47a22950d20fc62f6206d6a52644cdb59d19: Status 404 returned error can't find the container with id e66b49def86ef498fe55f2978dda47a22950d20fc62f6206d6a52644cdb59d19 Oct 30 23:44:33.348861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:33.348837 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2"] Oct 30 23:44:33.349983 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:33.349959 7570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:44:33.814151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:33.814050 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" event={"ID":"b6e10c50-ed06-4152-a626-64a1bc800d53","Type":"ContainerStarted","Data":"e66b49def86ef498fe55f2978dda47a22950d20fc62f6206d6a52644cdb59d19"} Oct 30 23:44:33.816235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:33.816201 7570 generic.go:356] "Generic (PLEG): container finished" podID="231f6ea2-fcf2-416c-adf5-8b195cb0df01" containerID="d2a4ba1529eb00d22b53bbdbb6fcdf460c9fb8070cc9948d8cfd49316f00de13" exitCode=0 Oct 30 23:44:33.816376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:33.816288 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" event={"ID":"231f6ea2-fcf2-416c-adf5-8b195cb0df01","Type":"ContainerDied","Data":"d2a4ba1529eb00d22b53bbdbb6fcdf460c9fb8070cc9948d8cfd49316f00de13"} Oct 30 23:44:34.605045 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:34.605011 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" Oct 30 23:44:34.828045 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:34.828014 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" event={"ID":"b6e10c50-ed06-4152-a626-64a1bc800d53","Type":"ContainerStarted","Data":"4b5c05eef61268a12336fdd3c58a349a61f58acd0706db40609b670360312d95"} Oct 30 23:44:34.828501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:34.828270 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:34.863362 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:34.862326 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" podStartSLOduration=2.213373533 podStartE2EDuration="2.862285206s" totalImagesPullingTime="648.911673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:34.85992361 +0000 UTC m=+1486.378924701" watchObservedRunningTime="2026-10-30 23:44:34.862285206 +0000 UTC m=+1486.381286301" Oct 30 23:44:35.052023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.051954 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:35.146367 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.146261 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231f6ea2-fcf2-416c-adf5-8b195cb0df01-kube-api-access-n8fdb\" (UniqueName: \"kubernetes.io/projected/231f6ea2-fcf2-416c-adf5-8b195cb0df01-kube-api-access-n8fdb\") pod \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\" (UID: \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\") " Oct 30 23:44:35.146544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.146445 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-openshift-ca\") pod \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\" (UID: \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\") " Oct 30 23:44:35.146794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.146771 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-xks-ca\") pod \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\" (UID: \"231f6ea2-fcf2-416c-adf5-8b195cb0df01\") " Oct 30 23:44:35.146892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.146796 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-openshift-ca" pod "231f6ea2-fcf2-416c-adf5-8b195cb0df01" (UID: "231f6ea2-fcf2-416c-adf5-8b195cb0df01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:35.147650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.147623 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:35.149346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.149322 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-xks-ca" pod "231f6ea2-fcf2-416c-adf5-8b195cb0df01" (UID: "231f6ea2-fcf2-416c-adf5-8b195cb0df01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:35.149822 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.149796 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231f6ea2-fcf2-416c-adf5-8b195cb0df01-kube-api-access-n8fdb" pod "231f6ea2-fcf2-416c-adf5-8b195cb0df01" (UID: "231f6ea2-fcf2-416c-adf5-8b195cb0df01"). InnerVolumeSpecName "kube-api-access-n8fdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:35.249337 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.249302 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8fdb\" (UniqueName: \"kubernetes.io/projected/231f6ea2-fcf2-416c-adf5-8b195cb0df01-kube-api-access-n8fdb\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:35.249337 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.249335 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231f6ea2-fcf2-416c-adf5-8b195cb0df01-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:35.838128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.838096 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" Oct 30 23:44:35.838128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.838106 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf" event={"ID":"231f6ea2-fcf2-416c-adf5-8b195cb0df01","Type":"ContainerDied","Data":"df517b7d8186ed3b431fd7846f47e02c84b76c54e7652e69758ac33c636e2916"} Oct 30 23:44:35.838610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:35.838180 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df517b7d8186ed3b431fd7846f47e02c84b76c54e7652e69758ac33c636e2916" Oct 30 23:44:36.026248 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:36.026213 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n"] Oct 30 23:44:36.031173 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:36.031141 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579e0042-7247n"] Oct 30 23:44:36.605897 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:36.605866 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2e7efb-89d5-4338-9a28-501fcc8b18dc" path="/var/lib/kubelet/pods/8a2e7efb-89d5-4338-9a28-501fcc8b18dc/volumes" Oct 30 23:44:39.028069 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:39.028041 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf"] Oct 30 23:44:39.032914 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:39.032894 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98cee74f-mqcpf"] Oct 30 23:44:40.546665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.546628 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff"] Oct 30 23:44:40.547902 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.547880 7570 cpu_manager.go:414] "containerMap: removing container" podUID="231f6ea2-fcf2-416c-adf5-8b195cb0df01" containerName="revoke-keys" Oct 30 23:44:40.548023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.547905 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231f6ea2-fcf2-416c-adf5-8b195cb0df01" containerName="revoke-keys" Oct 30 23:44:40.548175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.548161 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="231f6ea2-fcf2-416c-adf5-8b195cb0df01" containerName="revoke-keys" Oct 30 23:44:40.554068 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.554045 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:40.563419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.563376 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff"] Oct 30 23:44:40.568206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.568178 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:44:40.568350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.568236 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:44:40.606256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.606226 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8739891-ce63-4955-8382-74ceddeb4fc5" path="/var/lib/kubelet/pods/c8739891-ce63-4955-8382-74ceddeb4fc5/volumes" Oct 30 23:44:40.686729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.686698 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cz4g\" (UniqueName: \"kubernetes.io/projected/1286579b-ed48-4aca-a0a1-e0c8943f93b7-kube-api-access-6cz4g\") pod \"maas-api-revoke-keys-e2e-migrate-6cad1110-flvff\" (UID: \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:40.686977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.686952 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6cad1110-flvff\" (UID: \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:40.687215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.687189 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6cad1110-flvff\" (UID: \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:40.788326 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.788293 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6cad1110-flvff\" (UID: \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:40.788528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.788394 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cz4g\" (UniqueName: \"kubernetes.io/projected/1286579b-ed48-4aca-a0a1-e0c8943f93b7-kube-api-access-6cz4g\") pod \"maas-api-revoke-keys-e2e-migrate-6cad1110-flvff\" (UID: \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:40.788528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.788476 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6cad1110-flvff\" (UID: \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:40.788968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.788948 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6cad1110-flvff\" (UID: \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:40.791063 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.791042 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6cad1110-flvff\" (UID: \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:40.801943 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.801893 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cz4g\" (UniqueName: \"kubernetes.io/projected/1286579b-ed48-4aca-a0a1-e0c8943f93b7-kube-api-access-6cz4g\") pod \"maas-api-revoke-keys-e2e-migrate-6cad1110-flvff\" (UID: \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:40.844567 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.844542 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:44:40.894473 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:40.894437 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:41.197026 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:44:41.196993 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1286579b_ed48_4aca_a0a1_e0c8943f93b7.slice/crio-69d5c5ee4a82714f7b995e7018721331bbf0f62676a2c90ea3e110d82d9092dc WatchSource:0}: Error finding container 69d5c5ee4a82714f7b995e7018721331bbf0f62676a2c90ea3e110d82d9092dc: Status 404 returned error can't find the container with id 69d5c5ee4a82714f7b995e7018721331bbf0f62676a2c90ea3e110d82d9092dc Oct 30 23:44:41.197668 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:41.197647 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff"] Oct 30 23:44:41.927971 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:41.927933 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" event={"ID":"1286579b-ed48-4aca-a0a1-e0c8943f93b7","Type":"ContainerStarted","Data":"174c565459f7356c3e6849f813386e15e436907c1a1ee5ecb6a574c8d2eb2626"} Oct 30 23:44:41.928500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:41.927993 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" event={"ID":"1286579b-ed48-4aca-a0a1-e0c8943f93b7","Type":"ContainerStarted","Data":"69d5c5ee4a82714f7b995e7018721331bbf0f62676a2c90ea3e110d82d9092dc"} Oct 30 23:44:41.951499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:41.951432 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" podStartSLOduration=1.951418541 podStartE2EDuration="1.951418541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:41.949643491 +0000 UTC m=+1493.468644597" watchObservedRunningTime="2026-10-30 23:44:41.951418541 +0000 UTC m=+1493.470419630" Oct 30 23:44:42.940544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:42.940505 7570 generic.go:356] "Generic (PLEG): container finished" podID="1286579b-ed48-4aca-a0a1-e0c8943f93b7" containerID="174c565459f7356c3e6849f813386e15e436907c1a1ee5ecb6a574c8d2eb2626" exitCode=0 Oct 30 23:44:42.940946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:42.940592 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" event={"ID":"1286579b-ed48-4aca-a0a1-e0c8943f93b7","Type":"ContainerDied","Data":"174c565459f7356c3e6849f813386e15e436907c1a1ee5ecb6a574c8d2eb2626"} Oct 30 23:44:44.141312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.141285 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:44.150668 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.150638 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1286579b-ed48-4aca-a0a1-e0c8943f93b7-kube-api-access-6cz4g\" (UniqueName: \"kubernetes.io/projected/1286579b-ed48-4aca-a0a1-e0c8943f93b7-kube-api-access-6cz4g\") pod \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\" (UID: \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\") " Oct 30 23:44:44.150866 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.150845 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-xks-ca\") pod \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\" (UID: \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\") " Oct 30 23:44:44.150952 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.150922 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-openshift-ca\") pod \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\" (UID: \"1286579b-ed48-4aca-a0a1-e0c8943f93b7\") " Oct 30 23:44:44.151975 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.151346 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-openshift-ca" pod "1286579b-ed48-4aca-a0a1-e0c8943f93b7" (UID: "1286579b-ed48-4aca-a0a1-e0c8943f93b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:44.151975 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.151819 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:44.153259 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.153231 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1286579b-ed48-4aca-a0a1-e0c8943f93b7-kube-api-access-6cz4g" pod "1286579b-ed48-4aca-a0a1-e0c8943f93b7" (UID: "1286579b-ed48-4aca-a0a1-e0c8943f93b7"). InnerVolumeSpecName "kube-api-access-6cz4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:44.153352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.153302 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-xks-ca" pod "1286579b-ed48-4aca-a0a1-e0c8943f93b7" (UID: "1286579b-ed48-4aca-a0a1-e0c8943f93b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:44.253759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.253670 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1286579b-ed48-4aca-a0a1-e0c8943f93b7-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:44.253759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.253711 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cz4g\" (UniqueName: \"kubernetes.io/projected/1286579b-ed48-4aca-a0a1-e0c8943f93b7-kube-api-access-6cz4g\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:44.960544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.960444 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" Oct 30 23:44:44.960544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.960497 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff" event={"ID":"1286579b-ed48-4aca-a0a1-e0c8943f93b7","Type":"ContainerDied","Data":"69d5c5ee4a82714f7b995e7018721331bbf0f62676a2c90ea3e110d82d9092dc"} Oct 30 23:44:44.960728 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:44.960550 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69d5c5ee4a82714f7b995e7018721331bbf0f62676a2c90ea3e110d82d9092dc" Oct 30 23:44:45.031666 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.031636 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq"] Oct 30 23:44:45.036666 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.036640 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69287be-phqlq"] Oct 30 23:44:45.348066 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.348033 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq"] Oct 30 23:44:45.349095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.349066 7570 cpu_manager.go:414] "containerMap: removing container" podUID="1286579b-ed48-4aca-a0a1-e0c8943f93b7" containerName="revoke-keys" Oct 30 23:44:45.349095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.349089 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1286579b-ed48-4aca-a0a1-e0c8943f93b7" containerName="revoke-keys" Oct 30 23:44:45.349287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.349277 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1286579b-ed48-4aca-a0a1-e0c8943f93b7" containerName="revoke-keys" Oct 30 23:44:45.353566 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.353549 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:45.361390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.361360 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:44:45.361611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.361592 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:44:45.369981 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.369957 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq"] Oct 30 23:44:45.388905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.388877 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rkks\" (UniqueName: \"kubernetes.io/projected/06f47b10-565a-4118-8ee8-b51cea9f4a7f-kube-api-access-7rkks\") pod \"maas-api-revoke-keys-e2e-derive-ab09c434-smjpq\" (UID: \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:45.389074 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.389054 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ab09c434-smjpq\" (UID: \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:45.389173 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.389155 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ab09c434-smjpq\" (UID: \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:45.490950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.490919 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rkks\" (UniqueName: \"kubernetes.io/projected/06f47b10-565a-4118-8ee8-b51cea9f4a7f-kube-api-access-7rkks\") pod \"maas-api-revoke-keys-e2e-derive-ab09c434-smjpq\" (UID: \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:45.491150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.491045 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ab09c434-smjpq\" (UID: \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:45.491150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.491123 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ab09c434-smjpq\" (UID: \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:45.491855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.491838 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ab09c434-smjpq\" (UID: \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:45.493693 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.493677 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ab09c434-smjpq\" (UID: \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:45.504220 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.504194 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rkks\" (UniqueName: \"kubernetes.io/projected/06f47b10-565a-4118-8ee8-b51cea9f4a7f-kube-api-access-7rkks\") pod \"maas-api-revoke-keys-e2e-derive-ab09c434-smjpq\" (UID: \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:45.672507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.672391 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:45.999823 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:45.998552 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq"] Oct 30 23:44:46.605550 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:46.605518 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc200d5-c627-4beb-959c-0f0a4aaa60c1" path="/var/lib/kubelet/pods/ffc200d5-c627-4beb-959c-0f0a4aaa60c1/volumes" Oct 30 23:44:47.029453 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:47.029423 7570 generic.go:356] "Generic (PLEG): container finished" podID="06f47b10-565a-4118-8ee8-b51cea9f4a7f" containerID="6c9e0a88b09e666e45a7aec9871522c3ef1d4d52d36d38765e806d3e60a62489" exitCode=0 Oct 30 23:44:47.029644 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:47.029520 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" event={"ID":"06f47b10-565a-4118-8ee8-b51cea9f4a7f","Type":"ContainerDied","Data":"6c9e0a88b09e666e45a7aec9871522c3ef1d4d52d36d38765e806d3e60a62489"} Oct 30 23:44:47.029644 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:47.029575 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" event={"ID":"06f47b10-565a-4118-8ee8-b51cea9f4a7f","Type":"ContainerStarted","Data":"d0d3c8d426170a239dcb730cc9b8be7663a51adc8b0a7fccef2a2e8653c461bd"} Oct 30 23:44:48.033845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.033817 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc"] Oct 30 23:44:48.043915 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.043893 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1224a90c-bkrkc"] Oct 30 23:44:48.239619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.239591 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:48.340596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.340573 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-xks-ca\") pod \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\" (UID: \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\") " Oct 30 23:44:48.340736 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.340721 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-openshift-ca\") pod \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\" (UID: \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\") " Oct 30 23:44:48.341014 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.340990 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f47b10-565a-4118-8ee8-b51cea9f4a7f-kube-api-access-7rkks\" (UniqueName: \"kubernetes.io/projected/06f47b10-565a-4118-8ee8-b51cea9f4a7f-kube-api-access-7rkks\") pod \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\" (UID: \"06f47b10-565a-4118-8ee8-b51cea9f4a7f\") " Oct 30 23:44:48.341153 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.341098 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-openshift-ca" pod "06f47b10-565a-4118-8ee8-b51cea9f4a7f" (UID: "06f47b10-565a-4118-8ee8-b51cea9f4a7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:48.341561 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.341544 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:48.342859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.342835 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-xks-ca" pod "06f47b10-565a-4118-8ee8-b51cea9f4a7f" (UID: "06f47b10-565a-4118-8ee8-b51cea9f4a7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:48.343257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.343229 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f47b10-565a-4118-8ee8-b51cea9f4a7f-kube-api-access-7rkks" pod "06f47b10-565a-4118-8ee8-b51cea9f4a7f" (UID: "06f47b10-565a-4118-8ee8-b51cea9f4a7f"). InnerVolumeSpecName "kube-api-access-7rkks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:48.443599 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.443573 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rkks\" (UniqueName: \"kubernetes.io/projected/06f47b10-565a-4118-8ee8-b51cea9f4a7f-kube-api-access-7rkks\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:48.443694 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.443604 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f47b10-565a-4118-8ee8-b51cea9f4a7f-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:44:48.608842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:48.608766 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa07e4b-bd11-4810-af05-c8d358136fe9" path="/var/lib/kubelet/pods/2aa07e4b-bd11-4810-af05-c8d358136fe9/volumes" Oct 30 23:44:49.047995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:49.047911 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" Oct 30 23:44:49.047995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:49.047935 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq" event={"ID":"06f47b10-565a-4118-8ee8-b51cea9f4a7f","Type":"ContainerDied","Data":"d0d3c8d426170a239dcb730cc9b8be7663a51adc8b0a7fccef2a2e8653c461bd"} Oct 30 23:44:49.048637 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:49.048005 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0d3c8d426170a239dcb730cc9b8be7663a51adc8b0a7fccef2a2e8653c461bd" Oct 30 23:44:50.042995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:50.042906 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q"] Oct 30 23:44:50.047873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:50.047843 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659ff0c5-5gc8q"] Oct 30 23:44:50.604925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:50.604893 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2cbe38-7fc7-4274-8834-375b73af158a" path="/var/lib/kubelet/pods/0a2cbe38-7fc7-4274-8834-375b73af158a/volumes" Oct 30 23:44:54.128032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:54.127996 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff"] Oct 30 23:44:54.133526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:54.133500 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cad1110-flvff"] Oct 30 23:44:54.619741 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:54.619709 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1286579b-ed48-4aca-a0a1-e0c8943f93b7" path="/var/lib/kubelet/pods/1286579b-ed48-4aca-a0a1-e0c8943f93b7/volumes" Oct 30 23:44:55.511083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:44:55.511053 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb"] Oct 30 23:44:59.419229 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:44:59.419192 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:45:00.079994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.079930 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc"] Oct 30 23:45:00.086848 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.086819 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f8ebb2-rtphc"] Oct 30 23:45:00.182719 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.182680 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t"] Oct 30 23:45:00.183882 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.183860 7570 cpu_manager.go:414] "containerMap: removing container" podUID="06f47b10-565a-4118-8ee8-b51cea9f4a7f" containerName="revoke-keys" Oct 30 23:45:00.183951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.183885 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f47b10-565a-4118-8ee8-b51cea9f4a7f" containerName="revoke-keys" Oct 30 23:45:00.184168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.184156 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f47b10-565a-4118-8ee8-b51cea9f4a7f" containerName="revoke-keys" Oct 30 23:45:00.191127 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.191100 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7"] Oct 30 23:45:00.191271 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.191213 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" Oct 30 23:45:00.201402 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.196502 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" Oct 30 23:45:00.211888 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.209560 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7"] Oct 30 23:45:00.215546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.215518 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t"] Oct 30 23:45:00.240383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.240351 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l"] Oct 30 23:45:00.255094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.253008 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" Oct 30 23:45:00.269406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.269364 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8l82\" (UniqueName: \"kubernetes.io/projected/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-kube-api-access-x8l82\") pod \"maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7\" (UID: \"50c5e26f-8eb4-4827-9238-e126ec8cfbf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" Oct 30 23:45:00.269594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.269444 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ba48f4-89d7-4316-9c9e-62dbe0850f81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t\" (UID: \"63ba48f4-89d7-4316-9c9e-62dbe0850f81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" Oct 30 23:45:00.269594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.269550 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq7nr\" (UniqueName: \"kubernetes.io/projected/63ba48f4-89d7-4316-9c9e-62dbe0850f81-kube-api-access-lq7nr\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t\" (UID: \"63ba48f4-89d7-4316-9c9e-62dbe0850f81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" Oct 30 23:45:00.269715 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.269668 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7\" (UID: \"50c5e26f-8eb4-4827-9238-e126ec8cfbf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" Oct 30 23:45:00.275351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.275302 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k"] Oct 30 23:45:00.283875 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.283836 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" Oct 30 23:45:00.290163 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.290137 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l"] Oct 30 23:45:00.296582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.296556 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k"] Oct 30 23:45:00.298564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.298540 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:45:00.371451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.371360 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfznc\" (UniqueName: \"kubernetes.io/projected/3527398b-e06e-4bba-aafc-a8047faaedcf-kube-api-access-dfznc\") pod \"maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k\" (UID: \"3527398b-e06e-4bba-aafc-a8047faaedcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" Oct 30 23:45:00.371626 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.371520 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3527398b-e06e-4bba-aafc-a8047faaedcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k\" (UID: \"3527398b-e06e-4bba-aafc-a8047faaedcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" Oct 30 23:45:00.371626 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.371582 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8l82\" (UniqueName: \"kubernetes.io/projected/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-kube-api-access-x8l82\") pod \"maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7\" (UID: \"50c5e26f-8eb4-4827-9238-e126ec8cfbf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" Oct 30 23:45:00.371749 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.371647 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ba48f4-89d7-4316-9c9e-62dbe0850f81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t\" (UID: \"63ba48f4-89d7-4316-9c9e-62dbe0850f81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" Oct 30 23:45:00.371749 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.371714 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq7nr\" (UniqueName: \"kubernetes.io/projected/63ba48f4-89d7-4316-9c9e-62dbe0850f81-kube-api-access-lq7nr\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t\" (UID: \"63ba48f4-89d7-4316-9c9e-62dbe0850f81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" Oct 30 23:45:00.371852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.371773 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8kp4\" (UniqueName: \"kubernetes.io/projected/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-kube-api-access-w8kp4\") pod \"maas-api-key-cleanup-29890065-g6j5l\" (UID: \"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" Oct 30 23:45:00.371852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.371845 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7\" (UID: \"50c5e26f-8eb4-4827-9238-e126ec8cfbf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" Oct 30 23:45:00.372070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.372048 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890065-g6j5l\" (UID: \"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" Oct 30 23:45:00.372752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.372722 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ba48f4-89d7-4316-9c9e-62dbe0850f81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t\" (UID: \"63ba48f4-89d7-4316-9c9e-62dbe0850f81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" Oct 30 23:45:00.372964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.372945 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7\" (UID: \"50c5e26f-8eb4-4827-9238-e126ec8cfbf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" Oct 30 23:45:00.384861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.384823 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8l82\" (UniqueName: \"kubernetes.io/projected/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-kube-api-access-x8l82\") pod \"maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7\" (UID: \"50c5e26f-8eb4-4827-9238-e126ec8cfbf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" Oct 30 23:45:00.385034 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.384977 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq7nr\" (UniqueName: \"kubernetes.io/projected/63ba48f4-89d7-4316-9c9e-62dbe0850f81-kube-api-access-lq7nr\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t\" (UID: \"63ba48f4-89d7-4316-9c9e-62dbe0850f81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" Oct 30 23:45:00.474151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.474115 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890065-g6j5l\" (UID: \"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" Oct 30 23:45:00.474649 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.474275 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfznc\" (UniqueName: \"kubernetes.io/projected/3527398b-e06e-4bba-aafc-a8047faaedcf-kube-api-access-dfznc\") pod \"maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k\" (UID: \"3527398b-e06e-4bba-aafc-a8047faaedcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" Oct 30 23:45:00.474649 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.474338 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3527398b-e06e-4bba-aafc-a8047faaedcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k\" (UID: \"3527398b-e06e-4bba-aafc-a8047faaedcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" Oct 30 23:45:00.474649 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.474521 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8kp4\" (UniqueName: \"kubernetes.io/projected/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-kube-api-access-w8kp4\") pod \"maas-api-key-cleanup-29890065-g6j5l\" (UID: \"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" Oct 30 23:45:00.474946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.474927 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890065-g6j5l\" (UID: \"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" Oct 30 23:45:00.475145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.475122 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3527398b-e06e-4bba-aafc-a8047faaedcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k\" (UID: \"3527398b-e06e-4bba-aafc-a8047faaedcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" Oct 30 23:45:00.487952 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.487926 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8kp4\" (UniqueName: \"kubernetes.io/projected/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-kube-api-access-w8kp4\") pod \"maas-api-key-cleanup-29890065-g6j5l\" (UID: \"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" Oct 30 23:45:00.488078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.487926 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfznc\" (UniqueName: \"kubernetes.io/projected/3527398b-e06e-4bba-aafc-a8047faaedcf-kube-api-access-dfznc\") pod \"maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k\" (UID: \"3527398b-e06e-4bba-aafc-a8047faaedcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" Oct 30 23:45:00.511613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.511576 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" podUID="a67c142d-eada-462c-a292-cd01ce48ed0e" containerName="payload-pre-processing" containerID="cri-o://ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640" gracePeriod=25 Oct 30 23:45:00.529405 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.529377 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" Oct 30 23:45:00.536580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.536556 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" Oct 30 23:45:00.580628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.580598 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" Oct 30 23:45:00.610652 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.610142 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" Oct 30 23:45:00.611834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.611814 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0ce57c-9e80-4409-9957-957d5efcddff" path="/var/lib/kubelet/pods/ac0ce57c-9e80-4409-9957-957d5efcddff/volumes" Oct 30 23:45:00.844134 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:45:00.839705 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ba48f4_89d7_4316_9c9e_62dbe0850f81.slice/crio-fe22daf8e9f110033c477233a4626889b7fbf92e5da9e192033351104d5af081 WatchSource:0}: Error finding container fe22daf8e9f110033c477233a4626889b7fbf92e5da9e192033351104d5af081: Status 404 returned error can't find the container with id fe22daf8e9f110033c477233a4626889b7fbf92e5da9e192033351104d5af081 Oct 30 23:45:00.844134 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.842695 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7"] Oct 30 23:45:00.847673 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:45:00.847643 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c5e26f_8eb4_4827_9238_e126ec8cfbf8.slice/crio-92737af6982ca1e28efdee86f730bf229bf6570df2131b98b9dca515c61dee6d WatchSource:0}: Error finding container 92737af6982ca1e28efdee86f730bf229bf6570df2131b98b9dca515c61dee6d: Status 404 returned error can't find the container with id 92737af6982ca1e28efdee86f730bf229bf6570df2131b98b9dca515c61dee6d Oct 30 23:45:00.852964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.852936 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t"] Oct 30 23:45:00.909531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.909216 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l"] Oct 30 23:45:00.931116 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.930780 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" Oct 30 23:45:00.966171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:00.966141 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k"] Oct 30 23:45:00.966700 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:45:00.966672 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3527398b_e06e_4bba_aafc_a8047faaedcf.slice/crio-765fce042bf90099427f19eca8515ad94c3c877e7bb199f75d2d4fdf2aa65302 WatchSource:0}: Error finding container 765fce042bf90099427f19eca8515ad94c3c877e7bb199f75d2d4fdf2aa65302: Status 404 returned error can't find the container with id 765fce042bf90099427f19eca8515ad94c3c877e7bb199f75d2d4fdf2aa65302 Oct 30 23:45:01.036559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.036523 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92"] Oct 30 23:45:01.042187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.042155 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd78f4b-6wx92"] Oct 30 23:45:01.093086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.093057 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67c142d-eada-462c-a292-cd01ce48ed0e-kube-api-access-bc6pb\" (UniqueName: \"kubernetes.io/projected/a67c142d-eada-462c-a292-cd01ce48ed0e-kube-api-access-bc6pb\") pod \"a67c142d-eada-462c-a292-cd01ce48ed0e\" (UID: \"a67c142d-eada-462c-a292-cd01ce48ed0e\") " Oct 30 23:45:01.093275 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.093160 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67c142d-eada-462c-a292-cd01ce48ed0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67c142d-eada-462c-a292-cd01ce48ed0e-plugins-config-volume\") pod \"a67c142d-eada-462c-a292-cd01ce48ed0e\" (UID: \"a67c142d-eada-462c-a292-cd01ce48ed0e\") " Oct 30 23:45:01.093643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.093618 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67c142d-eada-462c-a292-cd01ce48ed0e-plugins-config-volume" pod "a67c142d-eada-462c-a292-cd01ce48ed0e" (UID: "a67c142d-eada-462c-a292-cd01ce48ed0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:01.093855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.093842 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67c142d-eada-462c-a292-cd01ce48ed0e-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.187735 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.187643 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67c142d-eada-462c-a292-cd01ce48ed0e-kube-api-access-bc6pb" pod "a67c142d-eada-462c-a292-cd01ce48ed0e" (UID: "a67c142d-eada-462c-a292-cd01ce48ed0e"). InnerVolumeSpecName "kube-api-access-bc6pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:01.195834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.195800 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc6pb\" (UniqueName: \"kubernetes.io/projected/a67c142d-eada-462c-a292-cd01ce48ed0e-kube-api-access-bc6pb\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.277639 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.277601 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" event={"ID":"50c5e26f-8eb4-4827-9238-e126ec8cfbf8","Type":"ContainerStarted","Data":"92737af6982ca1e28efdee86f730bf229bf6570df2131b98b9dca515c61dee6d"} Oct 30 23:45:01.279638 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.279605 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" event={"ID":"63ba48f4-89d7-4316-9c9e-62dbe0850f81","Type":"ContainerStarted","Data":"fe22daf8e9f110033c477233a4626889b7fbf92e5da9e192033351104d5af081"} Oct 30 23:45:01.281364 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.281330 7570 generic.go:356] "Generic (PLEG): container finished" podID="a67c142d-eada-462c-a292-cd01ce48ed0e" containerID="ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640" exitCode=0 Oct 30 23:45:01.281502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.281404 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" Oct 30 23:45:01.281502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.281412 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" event={"ID":"a67c142d-eada-462c-a292-cd01ce48ed0e","Type":"ContainerDied","Data":"ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640"} Oct 30 23:45:01.281642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.281500 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb" event={"ID":"a67c142d-eada-462c-a292-cd01ce48ed0e","Type":"ContainerDied","Data":"6ce5de1e2d915c79740c6e6ae22cb2bb50b9eede49851700f23b8550cda01bae"} Oct 30 23:45:01.281642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.281537 7570 scope.go:122] "RemoveContainer" containerID="ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640" Oct 30 23:45:01.286631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.286124 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" event={"ID":"3527398b-e06e-4bba-aafc-a8047faaedcf","Type":"ContainerStarted","Data":"765fce042bf90099427f19eca8515ad94c3c877e7bb199f75d2d4fdf2aa65302"} Oct 30 23:45:01.289657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.289581 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" event={"ID":"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0","Type":"ContainerStarted","Data":"d34d76f1e6d03fb4344ee53e96ba891283de3bcd1ddb16cb9b04de825fbe1d8a"} Oct 30 23:45:01.342787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.340486 7570 scope.go:122] "RemoveContainer" containerID="ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640" Oct 30 23:45:01.354697 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:01.354655 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640\": container with ID starting with ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640 not found: ID does not exist" containerID="ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640" Oct 30 23:45:01.354951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.354923 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640"} err="failed to get container status \"ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640\": rpc error: code = NotFound desc = could not find container \"ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640\": container with ID starting with ee5fe9d4aa244f47fc2ae30083940473653d9d67b1b55040fe54455e14fb6640 not found: ID does not exist" Oct 30 23:45:01.465432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.465367 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb"] Oct 30 23:45:01.491681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:01.491293 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ab09c434-6888d6f86c-hnvwb"] Oct 30 23:45:02.056680 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.021536 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2"] Oct 30 23:45:02.056680 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.021817 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" podUID="b6e10c50-ed06-4152-a626-64a1bc800d53" containerName="maas-api" containerID="cri-o://4b5c05eef61268a12336fdd3c58a349a61f58acd0706db40609b670360312d95" gracePeriod=30 Oct 30 23:45:02.118181 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.098562 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k"] Oct 30 23:45:02.314566 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.314431 7570 generic.go:356] "Generic (PLEG): container finished" podID="3527398b-e06e-4bba-aafc-a8047faaedcf" containerID="0e504c028419cf57c8d48a1df7e29945a99253481399d71f79b6b59722d6837f" exitCode=6 Oct 30 23:45:02.315308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.314828 7570 scope.go:122] "RemoveContainer" containerID="0e504c028419cf57c8d48a1df7e29945a99253481399d71f79b6b59722d6837f" Oct 30 23:45:02.315525 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.315111 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" event={"ID":"3527398b-e06e-4bba-aafc-a8047faaedcf","Type":"ContainerDied","Data":"0e504c028419cf57c8d48a1df7e29945a99253481399d71f79b6b59722d6837f"} Oct 30 23:45:02.324498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.323292 7570 generic.go:356] "Generic (PLEG): container finished" podID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" containerID="34993afdfc0530181ef7b13c46237edbe1f24f89ebbaba2c0afff7ad71f5f85b" exitCode=60 Oct 30 23:45:02.324498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.323434 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" event={"ID":"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0","Type":"ContainerDied","Data":"34993afdfc0530181ef7b13c46237edbe1f24f89ebbaba2c0afff7ad71f5f85b"} Oct 30 23:45:02.324498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.324161 7570 scope.go:122] "RemoveContainer" containerID="34993afdfc0530181ef7b13c46237edbe1f24f89ebbaba2c0afff7ad71f5f85b" Oct 30 23:45:02.328798 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.328173 7570 generic.go:356] "Generic (PLEG): container finished" podID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" containerID="5bcb13e4d3e5d8082c702489c5f81abbd00ab53ebfaa139c9c03dd2d20f434a5" exitCode=60 Oct 30 23:45:02.328798 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.328529 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" event={"ID":"50c5e26f-8eb4-4827-9238-e126ec8cfbf8","Type":"ContainerDied","Data":"5bcb13e4d3e5d8082c702489c5f81abbd00ab53ebfaa139c9c03dd2d20f434a5"} Oct 30 23:45:02.339038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.331028 7570 generic.go:356] "Generic (PLEG): container finished" podID="b6e10c50-ed06-4152-a626-64a1bc800d53" containerID="4b5c05eef61268a12336fdd3c58a349a61f58acd0706db40609b670360312d95" exitCode=0 Oct 30 23:45:02.339038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.331136 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" event={"ID":"b6e10c50-ed06-4152-a626-64a1bc800d53","Type":"ContainerDied","Data":"4b5c05eef61268a12336fdd3c58a349a61f58acd0706db40609b670360312d95"} Oct 30 23:45:02.339038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.333403 7570 scope.go:122] "RemoveContainer" containerID="5bcb13e4d3e5d8082c702489c5f81abbd00ab53ebfaa139c9c03dd2d20f434a5" Oct 30 23:45:02.342515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.342409 7570 generic.go:356] "Generic (PLEG): container finished" podID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" containerID="f10bc13aee37bf8e7187a7e580bf80900167e9a1c8312815b979cc006c8fa54b" exitCode=60 Oct 30 23:45:02.342963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.342900 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" event={"ID":"63ba48f4-89d7-4316-9c9e-62dbe0850f81","Type":"ContainerDied","Data":"f10bc13aee37bf8e7187a7e580bf80900167e9a1c8312815b979cc006c8fa54b"} Oct 30 23:45:02.343065 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.343048 7570 scope.go:122] "RemoveContainer" containerID="f10bc13aee37bf8e7187a7e580bf80900167e9a1c8312815b979cc006c8fa54b" Oct 30 23:45:02.717043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.712366 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c202e0d-cc38-45d4-ad64-b953c14160ad" path="/var/lib/kubelet/pods/0c202e0d-cc38-45d4-ad64-b953c14160ad/volumes" Oct 30 23:45:02.717043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:02.713584 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67c142d-eada-462c-a292-cd01ce48ed0e" path="/var/lib/kubelet/pods/a67c142d-eada-462c-a292-cd01ce48ed0e/volumes" Oct 30 23:45:03.241496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.237413 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:45:03.577559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.577498 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn"] Oct 30 23:45:03.641607 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.641524 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee37093-f55nn"] Oct 30 23:45:03.730062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.672833 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-metrics-certs\") pod \"b6e10c50-ed06-4152-a626-64a1bc800d53\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " Oct 30 23:45:03.730062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.672927 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e10c50-ed06-4152-a626-64a1bc800d53-kube-api-access-hs4vb\" (UniqueName: \"kubernetes.io/projected/b6e10c50-ed06-4152-a626-64a1bc800d53-kube-api-access-hs4vb\") pod \"b6e10c50-ed06-4152-a626-64a1bc800d53\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " Oct 30 23:45:03.730062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.672958 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-maas-api-tls\") pod \"b6e10c50-ed06-4152-a626-64a1bc800d53\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " Oct 30 23:45:03.730062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.673254 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e10c50-ed06-4152-a626-64a1bc800d53-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e10c50-ed06-4152-a626-64a1bc800d53-openshift-service-ca\") pod \"b6e10c50-ed06-4152-a626-64a1bc800d53\" (UID: \"b6e10c50-ed06-4152-a626-64a1bc800d53\") " Oct 30 23:45:03.730062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.689817 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e10c50-ed06-4152-a626-64a1bc800d53-openshift-service-ca" pod "b6e10c50-ed06-4152-a626-64a1bc800d53" (UID: "b6e10c50-ed06-4152-a626-64a1bc800d53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:03.730062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.696761 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq"] Oct 30 23:45:03.730062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.716583 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ab09c434-smjpq"] Oct 30 23:45:03.791486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.784291 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" event={"ID":"b6e10c50-ed06-4152-a626-64a1bc800d53","Type":"ContainerDied","Data":"e66b49def86ef498fe55f2978dda47a22950d20fc62f6206d6a52644cdb59d19"} Oct 30 23:45:03.791486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.784370 7570 scope.go:122] "RemoveContainer" containerID="4b5c05eef61268a12336fdd3c58a349a61f58acd0706db40609b670360312d95" Oct 30 23:45:03.791486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.784613 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2" Oct 30 23:45:03.827212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.797051 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e10c50-ed06-4152-a626-64a1bc800d53-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:03.992547 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:03.967525 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p"] Oct 30 23:45:04.019682 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.019641 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" podUID="b0d61ecc-0548-44d5-8852-b00e3b5b19da" containerName="istio-proxy" containerID="cri-o://66c2f14fe59c02ee855dcdf9ed62ef6390cd8866673457cbcc77cdc852ad4f25" gracePeriod=30 Oct 30 23:45:04.078518 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.078406 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-maas-api-tls" pod "b6e10c50-ed06-4152-a626-64a1bc800d53" (UID: "b6e10c50-ed06-4152-a626-64a1bc800d53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:45:04.078820 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.078781 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e10c50-ed06-4152-a626-64a1bc800d53-kube-api-access-hs4vb" pod "b6e10c50-ed06-4152-a626-64a1bc800d53" (UID: "b6e10c50-ed06-4152-a626-64a1bc800d53"). InnerVolumeSpecName "kube-api-access-hs4vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:04.082001 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.081949 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-metrics-certs" pod "b6e10c50-ed06-4152-a626-64a1bc800d53" (UID: "b6e10c50-ed06-4152-a626-64a1bc800d53"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:45:04.152048 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.152014 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs4vb\" (UniqueName: \"kubernetes.io/projected/b6e10c50-ed06-4152-a626-64a1bc800d53-kube-api-access-hs4vb\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:04.152269 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.152250 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:04.152393 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.152379 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e10c50-ed06-4152-a626-64a1bc800d53-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:04.264251 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.264139 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2"] Oct 30 23:45:04.272832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.272802 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ab09c434-7c8cc45fdd-vf6f2"] Oct 30 23:45:04.478808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.475680 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz"] Oct 30 23:45:04.478808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.476490 7570 cpu_manager.go:414] "containerMap: removing container" podUID="b6e10c50-ed06-4152-a626-64a1bc800d53" containerName="maas-api" Oct 30 23:45:04.478808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.476506 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e10c50-ed06-4152-a626-64a1bc800d53" containerName="maas-api" Oct 30 23:45:04.478808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.476523 7570 cpu_manager.go:414] "containerMap: removing container" podUID="a67c142d-eada-462c-a292-cd01ce48ed0e" containerName="payload-pre-processing" Oct 30 23:45:04.478808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.476529 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67c142d-eada-462c-a292-cd01ce48ed0e" containerName="payload-pre-processing" Oct 30 23:45:04.478808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.476643 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67c142d-eada-462c-a292-cd01ce48ed0e" containerName="payload-pre-processing" Oct 30 23:45:04.478808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.476664 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e10c50-ed06-4152-a626-64a1bc800d53" containerName="maas-api" Oct 30 23:45:04.482801 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.482777 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.506778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.506754 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-6c5084-openshift-default-dockercfg-bblz7\"" Oct 30 23:45:04.565110 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.565071 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz"] Oct 30 23:45:04.651961 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.651919 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f47b10-565a-4118-8ee8-b51cea9f4a7f" path="/var/lib/kubelet/pods/06f47b10-565a-4118-8ee8-b51cea9f4a7f/volumes" Oct 30 23:45:04.652713 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.652691 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723bafe3-3781-4262-a5ea-ef0669d065d4" path="/var/lib/kubelet/pods/723bafe3-3781-4262-a5ea-ef0669d065d4/volumes" Oct 30 23:45:04.653623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.653604 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e10c50-ed06-4152-a626-64a1bc800d53" path="/var/lib/kubelet/pods/b6e10c50-ed06-4152-a626-64a1bc800d53/volumes" Oct 30 23:45:04.669803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.669765 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-podinfo\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.669945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.669809 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48390cfe-87a3-4b4e-9986-8adec60f6349-istiod-ca-cert\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.669945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.669848 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-data\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.669945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.669874 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-socket\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.669945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.669905 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-envoy\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.670237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.670057 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-certs\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.670237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.670091 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-token\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.670353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.670280 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-credential-socket\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.670353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.670343 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q87ln\" (UniqueName: \"kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-kube-api-access-q87ln\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.791884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.789534 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-credential-socket\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.791884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.789659 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87ln\" (UniqueName: \"kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-kube-api-access-q87ln\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.791884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.789819 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-podinfo\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.791884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.789849 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48390cfe-87a3-4b4e-9986-8adec60f6349-istiod-ca-cert\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.791884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.789905 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-data\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.791884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.789945 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-socket\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.791884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.789981 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-envoy\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.808870 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.808774 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-certs\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.809099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.809065 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-token\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.810178 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.810149 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-credential-socket\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.839864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.827823 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-socket\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.839864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.828277 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-certs\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.839864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.828706 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-data\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.839864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.829103 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48390cfe-87a3-4b4e-9986-8adec60f6349-istiod-ca-cert\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.848114 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.848044 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-podinfo\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.979486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.977606 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-envoy\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:04.989484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:04.982372 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-token\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:05.027735 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.010178 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q87ln\" (UniqueName: \"kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-kube-api-access-q87ln\") pod \"e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:05.061633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.061072 7570 generic.go:356] "Generic (PLEG): container finished" podID="3527398b-e06e-4bba-aafc-a8047faaedcf" containerID="edcaac841436aa02814ae3a3f515e1eaf43f4c19d04e6290755462552d9d8c16" exitCode=6 Oct 30 23:45:05.061633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.061202 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" event={"ID":"3527398b-e06e-4bba-aafc-a8047faaedcf","Type":"ContainerDied","Data":"edcaac841436aa02814ae3a3f515e1eaf43f4c19d04e6290755462552d9d8c16"} Oct 30 23:45:05.061633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.061259 7570 scope.go:122] "RemoveContainer" containerID="0e504c028419cf57c8d48a1df7e29945a99253481399d71f79b6b59722d6837f" Oct 30 23:45:05.171708 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.170416 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:05.268018 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.267980 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" event={"ID":"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0","Type":"ContainerStarted","Data":"0aee4ea0162e0ed913ea09a3d401fd0352a95b5f0406dfcd2b0dc09aafabf600"} Oct 30 23:45:05.268721 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.268628 7570 scope.go:122] "RemoveContainer" containerID="0aee4ea0162e0ed913ea09a3d401fd0352a95b5f0406dfcd2b0dc09aafabf600" Oct 30 23:45:05.269152 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:05.269111 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890065-g6j5l_odh-ai-gateway-infra(3f27f4b6-5a56-4f85-a449-9305b8aeb9f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" Oct 30 23:45:05.276036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.276012 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" event={"ID":"50c5e26f-8eb4-4827-9238-e126ec8cfbf8","Type":"ContainerStarted","Data":"e0a9440805edace5e87c13ce963a8f42c65f6749e1aec3ba5b5fa9634c83a30e"} Oct 30 23:45:05.276665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.276570 7570 scope.go:122] "RemoveContainer" containerID="e0a9440805edace5e87c13ce963a8f42c65f6749e1aec3ba5b5fa9634c83a30e" Oct 30 23:45:05.277095 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:05.277073 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7_odh-ai-gateway-infra(50c5e26f-8eb4-4827-9238-e126ec8cfbf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" Oct 30 23:45:05.284729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.284694 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" event={"ID":"63ba48f4-89d7-4316-9c9e-62dbe0850f81","Type":"ContainerStarted","Data":"e02ae36eb11ed08fea237361aece4bcb0a553c99298387341e6f41d451539aea"} Oct 30 23:45:05.285359 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.285230 7570 scope.go:122] "RemoveContainer" containerID="e02ae36eb11ed08fea237361aece4bcb0a553c99298387341e6f41d451539aea" Oct 30 23:45:05.289535 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:05.289509 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t_odh-ai-gateway-infra(63ba48f4-89d7-4316-9c9e-62dbe0850f81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" Oct 30 23:45:05.663957 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.655451 7570 patch_prober.go:29] interesting pod/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:45:05.663957 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:05.655547 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" podUID="b0d61ecc-0548-44d5-8852-b00e3b5b19da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:45:06.494238 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:06.494188 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:45:06.495370 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:06.495334 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:45:06.589281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:06.589247 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" Oct 30 23:45:06.744554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:06.741138 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3527398b-e06e-4bba-aafc-a8047faaedcf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3527398b-e06e-4bba-aafc-a8047faaedcf-maas-api-service-ca\") pod \"3527398b-e06e-4bba-aafc-a8047faaedcf\" (UID: \"3527398b-e06e-4bba-aafc-a8047faaedcf\") " Oct 30 23:45:06.744554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:06.741207 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3527398b-e06e-4bba-aafc-a8047faaedcf-kube-api-access-dfznc\" (UniqueName: \"kubernetes.io/projected/3527398b-e06e-4bba-aafc-a8047faaedcf-kube-api-access-dfznc\") pod \"3527398b-e06e-4bba-aafc-a8047faaedcf\" (UID: \"3527398b-e06e-4bba-aafc-a8047faaedcf\") " Oct 30 23:45:06.826118 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:06.826058 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3527398b-e06e-4bba-aafc-a8047faaedcf-maas-api-service-ca" pod "3527398b-e06e-4bba-aafc-a8047faaedcf" (UID: "3527398b-e06e-4bba-aafc-a8047faaedcf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:06.881655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:06.870188 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:45:06.881655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:06.870670 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:45:06.906489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:06.882549 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:45:06.908407 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:06.908359 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:45:06.908939 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:06.883616 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:45:06.910260 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:06.883918 7570 trace.go:236] Trace[1569457260]: "Calculate volume metrics of manifests for pod opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" (30-Oct-2026 23:45:05.873) (total time: 1010ms): Oct 30 23:45:06.910260 ip-10-0-137-81 kubenswrapper[7570]: Trace[1569457260]: [1.010394314s] [1.010394314s] END Oct 30 23:45:07.062925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.050536 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3527398b-e06e-4bba-aafc-a8047faaedcf-kube-api-access-dfznc" pod "3527398b-e06e-4bba-aafc-a8047faaedcf" (UID: "3527398b-e06e-4bba-aafc-a8047faaedcf"). InnerVolumeSpecName "kube-api-access-dfznc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:07.062925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.051559 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:45:07.062925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.051923 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:45:07.062925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.053423 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:45:07.124115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.124076 7570 generic.go:356] "Generic (PLEG): container finished" podID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" containerID="e02ae36eb11ed08fea237361aece4bcb0a553c99298387341e6f41d451539aea" exitCode=60 Oct 30 23:45:07.129486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.126173 7570 scope.go:122] "RemoveContainer" containerID="e02ae36eb11ed08fea237361aece4bcb0a553c99298387341e6f41d451539aea" Oct 30 23:45:07.154643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.154580 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3527398b-e06e-4bba-aafc-a8047faaedcf-maas-api-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:07.154917 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.154895 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfznc\" (UniqueName: \"kubernetes.io/projected/3527398b-e06e-4bba-aafc-a8047faaedcf-kube-api-access-dfznc\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:07.156626 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:07.156580 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t_odh-ai-gateway-infra(63ba48f4-89d7-4316-9c9e-62dbe0850f81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" Oct 30 23:45:07.161169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.161129 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" Oct 30 23:45:07.255650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.255604 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:45:07.397091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.397028 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:45:07.433490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.430125 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:45:07.459421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.459377 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:45:07.495629 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.495579 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:45:07.496613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.496578 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:45:07.600451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.600410 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:45:07.671484 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:07.671377 7570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.07s" Oct 30 23:45:07.695703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.695667 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:45:07.696571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.696522 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" event={"ID":"63ba48f4-89d7-4316-9c9e-62dbe0850f81","Type":"ContainerDied","Data":"e02ae36eb11ed08fea237361aece4bcb0a553c99298387341e6f41d451539aea"} Oct 30 23:45:07.696819 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.696800 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:45:07.696950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.696936 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k" event={"ID":"3527398b-e06e-4bba-aafc-a8047faaedcf","Type":"ContainerDied","Data":"765fce042bf90099427f19eca8515ad94c3c877e7bb199f75d2d4fdf2aa65302"} Oct 30 23:45:07.700566 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.700532 7570 scope.go:122] "RemoveContainer" containerID="f10bc13aee37bf8e7187a7e580bf80900167e9a1c8312815b979cc006c8fa54b" Oct 30 23:45:07.871820 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.871782 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz"] Oct 30 23:45:07.878410 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.878371 7570 generic.go:356] "Generic (PLEG): container finished" podID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" containerID="0aee4ea0162e0ed913ea09a3d401fd0352a95b5f0406dfcd2b0dc09aafabf600" exitCode=60 Oct 30 23:45:07.878571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.878516 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" event={"ID":"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0","Type":"ContainerDied","Data":"0aee4ea0162e0ed913ea09a3d401fd0352a95b5f0406dfcd2b0dc09aafabf600"} Oct 30 23:45:07.879186 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.879163 7570 scope.go:122] "RemoveContainer" containerID="0aee4ea0162e0ed913ea09a3d401fd0352a95b5f0406dfcd2b0dc09aafabf600" Oct 30 23:45:07.879612 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:07.879583 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890065-g6j5l_odh-ai-gateway-infra(3f27f4b6-5a56-4f85-a449-9305b8aeb9f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" Oct 30 23:45:07.918427 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.918385 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:45:07.958995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:07.958901 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:45:08.027280 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.027050 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:45:08.054689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.054637 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:45:08.055051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.055014 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:45:08.063299 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.063268 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:45:08.063582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.063353 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:45:08.063582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.063392 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:45:08.063582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.063453 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:45:08.064667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.064345 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:45:08.069107 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.069064 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:45:08.069255 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.069232 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:45:08.069320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.069283 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:45:08.109970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.088185 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": context deadline exceeded" Oct 30 23:45:08.109970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.091797 7570 generic.go:356] "Generic (PLEG): container finished" podID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" containerID="e0a9440805edace5e87c13ce963a8f42c65f6749e1aec3ba5b5fa9634c83a30e" exitCode=60 Oct 30 23:45:08.109970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.091935 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" event={"ID":"50c5e26f-8eb4-4827-9238-e126ec8cfbf8","Type":"ContainerDied","Data":"e0a9440805edace5e87c13ce963a8f42c65f6749e1aec3ba5b5fa9634c83a30e"} Oct 30 23:45:08.109970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.092450 7570 scope.go:122] "RemoveContainer" containerID="e0a9440805edace5e87c13ce963a8f42c65f6749e1aec3ba5b5fa9634c83a30e" Oct 30 23:45:08.109970 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:08.092900 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7_odh-ai-gateway-infra(50c5e26f-8eb4-4827-9238-e126ec8cfbf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" Oct 30 23:45:08.109970 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:08.103603 7570 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3527398b_e06e_4bba_aafc_a8047faaedcf.slice\": RecentStats: unable to find data in memory cache]" Oct 30 23:45:08.109970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.109873 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:45:08.109970 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:08.108452 7570 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3527398b_e06e_4bba_aafc_a8047faaedcf.slice\": RecentStats: unable to find data in memory cache]" Oct 30 23:45:08.166075 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.166006 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:45:08.171213 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.170628 7570 scope.go:122] "RemoveContainer" containerID="edcaac841436aa02814ae3a3f515e1eaf43f4c19d04e6290755462552d9d8c16" Oct 30 23:45:08.171213 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.170932 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:45:08.255856 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.253782 7570 scope.go:122] "RemoveContainer" containerID="34993afdfc0530181ef7b13c46237edbe1f24f89ebbaba2c0afff7ad71f5f85b" Oct 30 23:45:08.261050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.261009 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:45:08.273132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.272679 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k"] Oct 30 23:45:08.279242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.279032 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-ab09c434-29890065-g5l2k"] Oct 30 23:45:08.300729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.300700 7570 scope.go:122] "RemoveContainer" containerID="5bcb13e4d3e5d8082c702489c5f81abbd00ab53ebfaa139c9c03dd2d20f434a5" Oct 30 23:45:08.650415 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:08.650387 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3527398b-e06e-4bba-aafc-a8047faaedcf" path="/var/lib/kubelet/pods/3527398b-e06e-4bba-aafc-a8047faaedcf/volumes" Oct 30 23:45:09.135253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:09.135223 7570 scope.go:122] "RemoveContainer" containerID="0aee4ea0162e0ed913ea09a3d401fd0352a95b5f0406dfcd2b0dc09aafabf600" Oct 30 23:45:09.135871 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:09.135829 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890065-g6j5l_odh-ai-gateway-infra(3f27f4b6-5a56-4f85-a449-9305b8aeb9f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" Oct 30 23:45:09.137555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:09.137522 7570 scope.go:122] "RemoveContainer" containerID="e0a9440805edace5e87c13ce963a8f42c65f6749e1aec3ba5b5fa9634c83a30e" Oct 30 23:45:09.137943 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:09.137916 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7_odh-ai-gateway-infra(50c5e26f-8eb4-4827-9238-e126ec8cfbf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" Oct 30 23:45:09.139396 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:09.139369 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" event={"ID":"48390cfe-87a3-4b4e-9986-8adec60f6349","Type":"ContainerStarted","Data":"8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738"} Oct 30 23:45:09.139538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:09.139517 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" event={"ID":"48390cfe-87a3-4b4e-9986-8adec60f6349","Type":"ContainerStarted","Data":"abcae9ccc790e314436463e92fd791146e561b7a1fa286837d5991a94913d592"} Oct 30 23:45:09.141542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:09.141521 7570 scope.go:122] "RemoveContainer" containerID="e02ae36eb11ed08fea237361aece4bcb0a553c99298387341e6f41d451539aea" Oct 30 23:45:09.141844 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:09.141821 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t_odh-ai-gateway-infra(63ba48f4-89d7-4316-9c9e-62dbe0850f81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" Oct 30 23:45:09.171111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:09.171074 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:09.178176 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:09.178145 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:09.197095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:09.197047 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" podStartSLOduration=5.197033036 podStartE2EDuration="5.197033036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:09.190253264 +0000 UTC m=+1520.709254354" watchObservedRunningTime="2026-10-30 23:45:09.197033036 +0000 UTC m=+1520.716034125" Oct 30 23:45:10.149952 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:10.149917 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:10.150764 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:10.150744 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:13.052604 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:13.052088 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb"] Oct 30 23:45:13.067483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:13.067124 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84a9e680-v2kdb"] Oct 30 23:45:14.608912 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:14.608873 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc2b2e0d-f349-455b-91f3-5186f9dfddeb" path="/var/lib/kubelet/pods/fc2b2e0d-f349-455b-91f3-5186f9dfddeb/volumes" Oct 30 23:45:15.595753 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:15.595724 7570 scope.go:122] "RemoveContainer" containerID="0aee4ea0162e0ed913ea09a3d401fd0352a95b5f0406dfcd2b0dc09aafabf600" Oct 30 23:45:16.297988 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:16.297895 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" event={"ID":"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0","Type":"ContainerStarted","Data":"c2c71747a08abb49dbe71af7540a241368eea3f18cb22b4e617a7e07af0a923a"} Oct 30 23:45:16.323922 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:16.323864 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" podStartSLOduration=16.323849442 podStartE2EDuration="16.323849442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:16.320828968 +0000 UTC m=+1527.839830060" watchObservedRunningTime="2026-10-30 23:45:16.323849442 +0000 UTC m=+1527.842850568" Oct 30 23:45:16.606540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:16.606511 7570 scope.go:122] "RemoveContainer" containerID="e0a9440805edace5e87c13ce963a8f42c65f6749e1aec3ba5b5fa9634c83a30e" Oct 30 23:45:17.311020 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:17.310941 7570 generic.go:356] "Generic (PLEG): container finished" podID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" containerID="c2c71747a08abb49dbe71af7540a241368eea3f18cb22b4e617a7e07af0a923a" exitCode=60 Oct 30 23:45:17.311520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:17.311014 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" event={"ID":"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0","Type":"ContainerDied","Data":"c2c71747a08abb49dbe71af7540a241368eea3f18cb22b4e617a7e07af0a923a"} Oct 30 23:45:17.311520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:17.311088 7570 scope.go:122] "RemoveContainer" containerID="0aee4ea0162e0ed913ea09a3d401fd0352a95b5f0406dfcd2b0dc09aafabf600" Oct 30 23:45:17.311635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:17.311620 7570 scope.go:122] "RemoveContainer" containerID="c2c71747a08abb49dbe71af7540a241368eea3f18cb22b4e617a7e07af0a923a" Oct 30 23:45:17.312170 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:17.312031 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890065-g6j5l_odh-ai-gateway-infra(3f27f4b6-5a56-4f85-a449-9305b8aeb9f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" Oct 30 23:45:17.313406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:17.313382 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" event={"ID":"50c5e26f-8eb4-4827-9238-e126ec8cfbf8","Type":"ContainerStarted","Data":"a5b1a725093db0ffdb4c687a30a0d1617d4d3fab2f907d8a4459e28ed48cdd3f"} Oct 30 23:45:17.339804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:17.339772 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l"] Oct 30 23:45:17.381517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:17.376563 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" podStartSLOduration=17.376543282 podStartE2EDuration="17.376543282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:17.375678486 +0000 UTC m=+1528.894679576" watchObservedRunningTime="2026-10-30 23:45:17.376543282 +0000 UTC m=+1528.895544374" Oct 30 23:45:17.595752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:17.595721 7570 scope.go:122] "RemoveContainer" containerID="e02ae36eb11ed08fea237361aece4bcb0a553c99298387341e6f41d451539aea" Oct 30 23:45:18.328726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.328684 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" event={"ID":"63ba48f4-89d7-4316-9c9e-62dbe0850f81","Type":"ContainerStarted","Data":"1dbf9a60739cfd051934d61ea6786d934be176b6819a1763a0c3bf37eab2f2ea"} Oct 30 23:45:18.332945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.332912 7570 generic.go:356] "Generic (PLEG): container finished" podID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" containerID="a5b1a725093db0ffdb4c687a30a0d1617d4d3fab2f907d8a4459e28ed48cdd3f" exitCode=60 Oct 30 23:45:18.333040 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.332982 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" event={"ID":"50c5e26f-8eb4-4827-9238-e126ec8cfbf8","Type":"ContainerDied","Data":"a5b1a725093db0ffdb4c687a30a0d1617d4d3fab2f907d8a4459e28ed48cdd3f"} Oct 30 23:45:18.333090 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.333051 7570 scope.go:122] "RemoveContainer" containerID="e0a9440805edace5e87c13ce963a8f42c65f6749e1aec3ba5b5fa9634c83a30e" Oct 30 23:45:18.333473 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.333449 7570 scope.go:122] "RemoveContainer" containerID="a5b1a725093db0ffdb4c687a30a0d1617d4d3fab2f907d8a4459e28ed48cdd3f" Oct 30 23:45:18.333887 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:18.333863 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7_odh-ai-gateway-infra(50c5e26f-8eb4-4827-9238-e126ec8cfbf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" Oct 30 23:45:18.348421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.348362 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" podStartSLOduration=18.34834944 podStartE2EDuration="18.34834944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:18.347215043 +0000 UTC m=+1529.866216135" watchObservedRunningTime="2026-10-30 23:45:18.34834944 +0000 UTC m=+1529.867350530" Oct 30 23:45:18.404285 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.404250 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7"] Oct 30 23:45:18.656787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.656747 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb"] Oct 30 23:45:18.657948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.657930 7570 cpu_manager.go:414] "containerMap: removing container" podUID="3527398b-e06e-4bba-aafc-a8047faaedcf" containerName="cleanup" Oct 30 23:45:18.658007 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.657952 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3527398b-e06e-4bba-aafc-a8047faaedcf" containerName="cleanup" Oct 30 23:45:18.658007 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.658001 7570 cpu_manager.go:414] "containerMap: removing container" podUID="3527398b-e06e-4bba-aafc-a8047faaedcf" containerName="cleanup" Oct 30 23:45:18.658073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.658010 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3527398b-e06e-4bba-aafc-a8047faaedcf" containerName="cleanup" Oct 30 23:45:18.658237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.658224 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3527398b-e06e-4bba-aafc-a8047faaedcf" containerName="cleanup" Oct 30 23:45:18.658274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.658248 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3527398b-e06e-4bba-aafc-a8047faaedcf" containerName="cleanup" Oct 30 23:45:18.684051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.684023 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb"] Oct 30 23:45:18.684241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.684108 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:18.692281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.692247 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-6c5084-serving-cert\"" Oct 30 23:45:18.785061 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.785030 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4af4419c-fffd-463e-9577-94bad299abc6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:18.889401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.785187 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52j88\" (UniqueName: \"kubernetes.io/projected/4af4419c-fffd-463e-9577-94bad299abc6-kube-api-access-52j88\") pod \"maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:18.889401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.785384 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:18.889401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.785426 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:18.904362 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.904325 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52j88\" (UniqueName: \"kubernetes.io/projected/4af4419c-fffd-463e-9577-94bad299abc6-kube-api-access-52j88\") pod \"maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:18.905003 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.904978 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:18.905743 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.905722 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:18.906633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.906605 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4af4419c-fffd-463e-9577-94bad299abc6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:18.907437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.907345 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4af4419c-fffd-463e-9577-94bad299abc6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:18.917535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.917502 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:18.928832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.921284 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52j88\" (UniqueName: \"kubernetes.io/projected/4af4419c-fffd-463e-9577-94bad299abc6-kube-api-access-52j88\") pod \"maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:18.928832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:18.922391 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:19.002716 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.002688 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" Oct 30 23:45:19.007739 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.007707 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-maas-api-service-ca\") pod \"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0\" (UID: \"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0\") " Oct 30 23:45:19.007873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.007836 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-kube-api-access-w8kp4\" (UniqueName: \"kubernetes.io/projected/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-kube-api-access-w8kp4\") pod \"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0\" (UID: \"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0\") " Oct 30 23:45:19.009035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.009007 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-maas-api-service-ca" pod "3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" (UID: "3f27f4b6-5a56-4f85-a449-9305b8aeb9f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:19.011267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.011248 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-kube-api-access-w8kp4" pod "3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" (UID: "3f27f4b6-5a56-4f85-a449-9305b8aeb9f0"). InnerVolumeSpecName "kube-api-access-w8kp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:19.066664 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.066637 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:19.109802 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.109769 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-maas-api-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.109802 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.109802 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8kp4\" (UniqueName: \"kubernetes.io/projected/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0-kube-api-access-w8kp4\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.365752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.365597 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t"] Oct 30 23:45:19.384369 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.384272 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb"] Oct 30 23:45:19.390891 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:45:19.390854 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af4419c_fffd_463e_9577_94bad299abc6.slice/crio-148966b0684dd6a7e73039ee4833512c2faa9fbf331d183d128cd76647e653ec WatchSource:0}: Error finding container 148966b0684dd6a7e73039ee4833512c2faa9fbf331d183d128cd76647e653ec: Status 404 returned error can't find the container with id 148966b0684dd6a7e73039ee4833512c2faa9fbf331d183d128cd76647e653ec Oct 30 23:45:19.406913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.406868 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" event={"ID":"3f27f4b6-5a56-4f85-a449-9305b8aeb9f0","Type":"ContainerDied","Data":"d34d76f1e6d03fb4344ee53e96ba891283de3bcd1ddb16cb9b04de825fbe1d8a"} Oct 30 23:45:19.407086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.406935 7570 scope.go:122] "RemoveContainer" containerID="c2c71747a08abb49dbe71af7540a241368eea3f18cb22b4e617a7e07af0a923a" Oct 30 23:45:19.407188 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.407084 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l" Oct 30 23:45:19.417223 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.417187 7570 generic.go:356] "Generic (PLEG): container finished" podID="b0d61ecc-0548-44d5-8852-b00e3b5b19da" containerID="66c2f14fe59c02ee855dcdf9ed62ef6390cd8866673457cbcc77cdc852ad4f25" exitCode=0 Oct 30 23:45:19.417368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.417348 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" event={"ID":"b0d61ecc-0548-44d5-8852-b00e3b5b19da","Type":"ContainerDied","Data":"66c2f14fe59c02ee855dcdf9ed62ef6390cd8866673457cbcc77cdc852ad4f25"} Oct 30 23:45:19.520902 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.520872 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l"] Oct 30 23:45:19.538476 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.538343 7570 generic.go:356] "Generic (PLEG): container finished" podID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" containerID="1dbf9a60739cfd051934d61ea6786d934be176b6819a1763a0c3bf37eab2f2ea" exitCode=60 Oct 30 23:45:19.538648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.538529 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" event={"ID":"63ba48f4-89d7-4316-9c9e-62dbe0850f81","Type":"ContainerDied","Data":"1dbf9a60739cfd051934d61ea6786d934be176b6819a1763a0c3bf37eab2f2ea"} Oct 30 23:45:19.538648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.538640 7570 scope.go:122] "RemoveContainer" containerID="e02ae36eb11ed08fea237361aece4bcb0a553c99298387341e6f41d451539aea" Oct 30 23:45:19.551580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.551270 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890065-g6j5l"] Oct 30 23:45:19.551580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.551546 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:45:19.631701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.631661 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-token\" (UniqueName: \"kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-token\") pod \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " Oct 30 23:45:19.631701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.631697 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istiod-ca-cert\") pod \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " Oct 30 23:45:19.631970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.631722 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-data\") pod \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " Oct 30 23:45:19.631970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.631815 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-podinfo\") pod \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " Oct 30 23:45:19.631970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.631928 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-kube-api-access-cgxr7\" (UniqueName: \"kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-kube-api-access-cgxr7\") pod \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " Oct 30 23:45:19.632212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.631975 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-socket\") pod \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " Oct 30 23:45:19.632212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.632128 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-envoy\") pod \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " Oct 30 23:45:19.632363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.632257 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-credential-socket\") pod \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " Oct 30 23:45:19.632363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.632329 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-certs\") pod \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\" (UID: \"b0d61ecc-0548-44d5-8852-b00e3b5b19da\") " Oct 30 23:45:19.632580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.632362 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-data" pod "b0d61ecc-0548-44d5-8852-b00e3b5b19da" (UID: "b0d61ecc-0548-44d5-8852-b00e3b5b19da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:19.632741 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.632645 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-socket" pod "b0d61ecc-0548-44d5-8852-b00e3b5b19da" (UID: "b0d61ecc-0548-44d5-8852-b00e3b5b19da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:19.632893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.632853 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-certs" pod "b0d61ecc-0548-44d5-8852-b00e3b5b19da" (UID: "b0d61ecc-0548-44d5-8852-b00e3b5b19da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:19.633286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.633260 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.633396 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.633381 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.633521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.633506 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.634446 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.634420 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-credential-socket" pod "b0d61ecc-0548-44d5-8852-b00e3b5b19da" (UID: "b0d61ecc-0548-44d5-8852-b00e3b5b19da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:19.636107 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.636089 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istiod-ca-cert" pod "b0d61ecc-0548-44d5-8852-b00e3b5b19da" (UID: "b0d61ecc-0548-44d5-8852-b00e3b5b19da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:19.638376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.637851 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-envoy" pod "b0d61ecc-0548-44d5-8852-b00e3b5b19da" (UID: "b0d61ecc-0548-44d5-8852-b00e3b5b19da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:19.638744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.638723 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-kube-api-access-cgxr7" pod "b0d61ecc-0548-44d5-8852-b00e3b5b19da" (UID: "b0d61ecc-0548-44d5-8852-b00e3b5b19da"). InnerVolumeSpecName "kube-api-access-cgxr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:19.638982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.638958 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-podinfo" pod "b0d61ecc-0548-44d5-8852-b00e3b5b19da" (UID: "b0d61ecc-0548-44d5-8852-b00e3b5b19da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:45:19.639225 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.639206 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-token" pod "b0d61ecc-0548-44d5-8852-b00e3b5b19da" (UID: "b0d61ecc-0548-44d5-8852-b00e3b5b19da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:19.714785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.714755 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" Oct 30 23:45:19.738314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.735243 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-kube-api-access-x8l82\" (UniqueName: \"kubernetes.io/projected/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-kube-api-access-x8l82\") pod \"50c5e26f-8eb4-4827-9238-e126ec8cfbf8\" (UID: \"50c5e26f-8eb4-4827-9238-e126ec8cfbf8\") " Oct 30 23:45:19.738314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.735531 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-maas-api-service-ca\") pod \"50c5e26f-8eb4-4827-9238-e126ec8cfbf8\" (UID: \"50c5e26f-8eb4-4827-9238-e126ec8cfbf8\") " Oct 30 23:45:19.738314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.736308 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.738314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.736328 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.738314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.736342 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.738314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.736357 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgxr7\" (UniqueName: \"kubernetes.io/projected/b0d61ecc-0548-44d5-8852-b00e3b5b19da-kube-api-access-cgxr7\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.738314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.736372 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.738314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.736386 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d61ecc-0548-44d5-8852-b00e3b5b19da-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.738314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.736763 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-maas-api-service-ca" pod "50c5e26f-8eb4-4827-9238-e126ec8cfbf8" (UID: "50c5e26f-8eb4-4827-9238-e126ec8cfbf8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:19.772706 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.772670 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-kube-api-access-x8l82" pod "50c5e26f-8eb4-4827-9238-e126ec8cfbf8" (UID: "50c5e26f-8eb4-4827-9238-e126ec8cfbf8"). InnerVolumeSpecName "kube-api-access-x8l82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:19.832689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.832661 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" Oct 30 23:45:19.837008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.836977 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ba48f4-89d7-4316-9c9e-62dbe0850f81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ba48f4-89d7-4316-9c9e-62dbe0850f81-maas-api-service-ca\") pod \"63ba48f4-89d7-4316-9c9e-62dbe0850f81\" (UID: \"63ba48f4-89d7-4316-9c9e-62dbe0850f81\") " Oct 30 23:45:19.837217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.837199 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ba48f4-89d7-4316-9c9e-62dbe0850f81-kube-api-access-lq7nr\" (UniqueName: \"kubernetes.io/projected/63ba48f4-89d7-4316-9c9e-62dbe0850f81-kube-api-access-lq7nr\") pod \"63ba48f4-89d7-4316-9c9e-62dbe0850f81\" (UID: \"63ba48f4-89d7-4316-9c9e-62dbe0850f81\") " Oct 30 23:45:19.837980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.837961 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-maas-api-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.838099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.837984 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8l82\" (UniqueName: \"kubernetes.io/projected/50c5e26f-8eb4-4827-9238-e126ec8cfbf8-kube-api-access-x8l82\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.838590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.838442 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ba48f4-89d7-4316-9c9e-62dbe0850f81-maas-api-service-ca" pod "63ba48f4-89d7-4316-9c9e-62dbe0850f81" (UID: "63ba48f4-89d7-4316-9c9e-62dbe0850f81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:19.840743 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.840719 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ba48f4-89d7-4316-9c9e-62dbe0850f81-kube-api-access-lq7nr" pod "63ba48f4-89d7-4316-9c9e-62dbe0850f81" (UID: "63ba48f4-89d7-4316-9c9e-62dbe0850f81"). InnerVolumeSpecName "kube-api-access-lq7nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:19.939183 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.939110 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq7nr\" (UniqueName: \"kubernetes.io/projected/63ba48f4-89d7-4316-9c9e-62dbe0850f81-kube-api-access-lq7nr\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.939183 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:19.939141 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ba48f4-89d7-4316-9c9e-62dbe0850f81-maas-api-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:20.550306 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.550275 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" event={"ID":"4af4419c-fffd-463e-9577-94bad299abc6","Type":"ContainerStarted","Data":"7109ce1e7cdee0e506b5064f13410fff49b1fa11c0d3b80c387c43b8850f7c5f"} Oct 30 23:45:20.550792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.550350 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:20.550792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.550378 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" event={"ID":"4af4419c-fffd-463e-9577-94bad299abc6","Type":"ContainerStarted","Data":"148966b0684dd6a7e73039ee4833512c2faa9fbf331d183d128cd76647e653ec"} Oct 30 23:45:20.552839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.552818 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" Oct 30 23:45:20.552963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.552869 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7" event={"ID":"50c5e26f-8eb4-4827-9238-e126ec8cfbf8","Type":"ContainerDied","Data":"92737af6982ca1e28efdee86f730bf229bf6570df2131b98b9dca515c61dee6d"} Oct 30 23:45:20.552963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.552923 7570 scope.go:122] "RemoveContainer" containerID="a5b1a725093db0ffdb4c687a30a0d1617d4d3fab2f907d8a4459e28ed48cdd3f" Oct 30 23:45:20.554730 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.554705 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" event={"ID":"b0d61ecc-0548-44d5-8852-b00e3b5b19da","Type":"ContainerDied","Data":"24ebbb76c0feb9ff66f9a954167251944c5a15b1857ed623d3918d4b6aa13089"} Oct 30 23:45:20.554845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.554806 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p" Oct 30 23:45:20.556246 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.556228 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" Oct 30 23:45:20.556246 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.556246 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t" event={"ID":"63ba48f4-89d7-4316-9c9e-62dbe0850f81","Type":"ContainerDied","Data":"fe22daf8e9f110033c477233a4626889b7fbf92e5da9e192033351104d5af081"} Oct 30 23:45:20.567110 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.567096 7570 scope.go:122] "RemoveContainer" containerID="66c2f14fe59c02ee855dcdf9ed62ef6390cd8866673457cbcc77cdc852ad4f25" Oct 30 23:45:20.575226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.575188 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podStartSLOduration=2.038874745 podStartE2EDuration="2.575175922s" totalImagesPullingTime="536.301177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:20.570038787 +0000 UTC m=+1532.089039878" watchObservedRunningTime="2026-10-30 23:45:20.575175922 +0000 UTC m=+1532.094177019" Oct 30 23:45:20.579243 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.579134 7570 scope.go:122] "RemoveContainer" containerID="1dbf9a60739cfd051934d61ea6786d934be176b6819a1763a0c3bf37eab2f2ea" Oct 30 23:45:20.590472 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.590433 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p"] Oct 30 23:45:20.607828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.607804 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" path="/var/lib/kubelet/pods/3f27f4b6-5a56-4f85-a449-9305b8aeb9f0/volumes" Oct 30 23:45:20.608218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.608203 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ab09c434-openshift-default-b4f6784cc-b5q8p"] Oct 30 23:45:20.608271 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.608232 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7"] Oct 30 23:45:20.611088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.611074 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890065-8njf7"] Oct 30 23:45:20.626580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.626556 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t"] Oct 30 23:45:20.630160 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:20.630143 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890065-8lh8t"] Oct 30 23:45:22.605374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:22.605338 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" path="/var/lib/kubelet/pods/50c5e26f-8eb4-4827-9238-e126ec8cfbf8/volumes" Oct 30 23:45:22.605969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:22.605949 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" path="/var/lib/kubelet/pods/63ba48f4-89d7-4316-9c9e-62dbe0850f81/volumes" Oct 30 23:45:22.606547 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:22.606529 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d61ecc-0548-44d5-8852-b00e3b5b19da" path="/var/lib/kubelet/pods/b0d61ecc-0548-44d5-8852-b00e3b5b19da/volumes" Oct 30 23:45:23.027054 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:23.026972 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4"] Oct 30 23:45:23.031536 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:23.031513 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a268da2b-kgwf4"] Oct 30 23:45:24.603855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:24.603823 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e473a80f-5ef4-459c-9d88-db39f2393b04" path="/var/lib/kubelet/pods/e473a80f-5ef4-459c-9d88-db39f2393b04/volumes" Oct 30 23:45:26.613581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:26.613554 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:45:29.413627 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:29.413548 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:45:35.046739 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:35.045804 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf"] Oct 30 23:45:35.052602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:35.052574 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53b3fe8-j5rcf"] Oct 30 23:45:36.645998 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:36.645965 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf3579b-b533-4d34-b609-565c42d51336" path="/var/lib/kubelet/pods/bbf3579b-b533-4d34-b609-565c42d51336/volumes" Oct 30 23:45:38.570563 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.570522 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb"] Oct 30 23:45:38.571293 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571278 7570 cpu_manager.go:414] "containerMap: removing container" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" containerName="cleanup" Oct 30 23:45:38.571349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571296 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" containerName="cleanup" Oct 30 23:45:38.571349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571312 7570 cpu_manager.go:414] "containerMap: removing container" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" containerName="cleanup" Oct 30 23:45:38.571349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571330 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" containerName="cleanup" Oct 30 23:45:38.571349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571343 7570 cpu_manager.go:414] "containerMap: removing container" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" containerName="cleanup" Oct 30 23:45:38.571349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571347 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" containerName="cleanup" Oct 30 23:45:38.571539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571355 7570 cpu_manager.go:414] "containerMap: removing container" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" containerName="cleanup" Oct 30 23:45:38.571539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571360 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" containerName="cleanup" Oct 30 23:45:38.571539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571366 7570 cpu_manager.go:414] "containerMap: removing container" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" containerName="cleanup" Oct 30 23:45:38.571539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571371 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" containerName="cleanup" Oct 30 23:45:38.571539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571378 7570 cpu_manager.go:414] "containerMap: removing container" podUID="b0d61ecc-0548-44d5-8852-b00e3b5b19da" containerName="istio-proxy" Oct 30 23:45:38.571539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571382 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d61ecc-0548-44d5-8852-b00e3b5b19da" containerName="istio-proxy" Oct 30 23:45:38.571539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571417 7570 cpu_manager.go:414] "containerMap: removing container" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" containerName="cleanup" Oct 30 23:45:38.571539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571421 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" containerName="cleanup" Oct 30 23:45:38.571539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571452 7570 cpu_manager.go:414] "containerMap: removing container" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" containerName="cleanup" Oct 30 23:45:38.571539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571470 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" containerName="cleanup" Oct 30 23:45:38.571840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571577 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" containerName="cleanup" Oct 30 23:45:38.571840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571585 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" containerName="cleanup" Oct 30 23:45:38.571840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571591 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" containerName="cleanup" Oct 30 23:45:38.571840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571601 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" containerName="cleanup" Oct 30 23:45:38.571840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571606 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" containerName="cleanup" Oct 30 23:45:38.571840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571610 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" containerName="cleanup" Oct 30 23:45:38.571840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.571615 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d61ecc-0548-44d5-8852-b00e3b5b19da" containerName="istio-proxy" Oct 30 23:45:38.575551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.575535 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:45:38.598806 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.598779 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-a3e876-dockercfg-2tm45\"" Oct 30 23:45:38.612638 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.612615 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-a3e876\"" Oct 30 23:45:38.630826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.630805 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb"] Oct 30 23:45:38.632898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.632771 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b85vb\" (UniqueName: \"kubernetes.io/projected/d387eda1-c364-4443-ba0d-619368f57f28-kube-api-access-b85vb\") pod \"payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb\" (UID: \"d387eda1-c364-4443-ba0d-619368f57f28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:45:38.632898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.632840 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d387eda1-c364-4443-ba0d-619368f57f28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb\" (UID: \"d387eda1-c364-4443-ba0d-619368f57f28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:45:38.929605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.929558 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b85vb\" (UniqueName: \"kubernetes.io/projected/d387eda1-c364-4443-ba0d-619368f57f28-kube-api-access-b85vb\") pod \"payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb\" (UID: \"d387eda1-c364-4443-ba0d-619368f57f28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:45:38.931425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.931397 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d387eda1-c364-4443-ba0d-619368f57f28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb\" (UID: \"d387eda1-c364-4443-ba0d-619368f57f28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:45:38.932805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.932760 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d387eda1-c364-4443-ba0d-619368f57f28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb\" (UID: \"d387eda1-c364-4443-ba0d-619368f57f28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:45:38.966823 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:38.966796 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b85vb\" (UniqueName: \"kubernetes.io/projected/d387eda1-c364-4443-ba0d-619368f57f28-kube-api-access-b85vb\") pod \"payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb\" (UID: \"d387eda1-c364-4443-ba0d-619368f57f28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:45:39.223873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:39.223772 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:45:39.616740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:39.616712 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb"] Oct 30 23:45:39.979748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:39.979651 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" event={"ID":"d387eda1-c364-4443-ba0d-619368f57f28","Type":"ContainerStarted","Data":"23e7358568ccd8720464b8ec3cfa6637e23cbe8a66959e3052dd5bbddf4c7a4a"} Oct 30 23:45:39.979748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:39.979706 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" event={"ID":"d387eda1-c364-4443-ba0d-619368f57f28","Type":"ContainerStarted","Data":"5d606fe2faaed4cf389dca4762738730526be205c7b2f186efced4a8f0e0dbd3"} Oct 30 23:45:39.979748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:39.979732 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:45:40.012299 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:40.012241 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" podStartSLOduration=2.012218932 podStartE2EDuration="2.012218932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:39.999181929 +0000 UTC m=+1551.518183020" watchObservedRunningTime="2026-10-30 23:45:40.012218932 +0000 UTC m=+1551.531220023" Oct 30 23:45:41.148079 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.148041 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk"] Oct 30 23:45:41.148972 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.148950 7570 cpu_manager.go:414] "containerMap: removing container" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" containerName="cleanup" Oct 30 23:45:41.149083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.148973 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" containerName="cleanup" Oct 30 23:45:41.149083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.149010 7570 cpu_manager.go:414] "containerMap: removing container" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" containerName="cleanup" Oct 30 23:45:41.149083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.149017 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" containerName="cleanup" Oct 30 23:45:41.149262 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.149247 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f27f4b6-5a56-4f85-a449-9305b8aeb9f0" containerName="cleanup" Oct 30 23:45:41.149320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.149275 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c5e26f-8eb4-4827-9238-e126ec8cfbf8" containerName="cleanup" Oct 30 23:45:41.149320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.149287 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ba48f4-89d7-4316-9c9e-62dbe0850f81" containerName="cleanup" Oct 30 23:45:41.153149 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.153130 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:41.158723 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.158700 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:45:41.159317 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.158746 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:45:41.161356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.161336 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk"] Oct 30 23:45:41.183823 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.183797 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk\" (UID: \"08111205-a51a-422b-9ec4-16555a5142a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:41.183936 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.183834 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk\" (UID: \"08111205-a51a-422b-9ec4-16555a5142a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:41.183996 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.183948 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttmfn\" (UniqueName: \"kubernetes.io/projected/08111205-a51a-422b-9ec4-16555a5142a7-kube-api-access-ttmfn\") pod \"maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk\" (UID: \"08111205-a51a-422b-9ec4-16555a5142a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:41.285797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.285764 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk\" (UID: \"08111205-a51a-422b-9ec4-16555a5142a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:41.285953 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.285811 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk\" (UID: \"08111205-a51a-422b-9ec4-16555a5142a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:41.285953 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.285945 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttmfn\" (UniqueName: \"kubernetes.io/projected/08111205-a51a-422b-9ec4-16555a5142a7-kube-api-access-ttmfn\") pod \"maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk\" (UID: \"08111205-a51a-422b-9ec4-16555a5142a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:41.286613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.286573 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk\" (UID: \"08111205-a51a-422b-9ec4-16555a5142a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:41.288553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.288531 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk\" (UID: \"08111205-a51a-422b-9ec4-16555a5142a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:41.304196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.304164 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttmfn\" (UniqueName: \"kubernetes.io/projected/08111205-a51a-422b-9ec4-16555a5142a7-kube-api-access-ttmfn\") pod \"maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk\" (UID: \"08111205-a51a-422b-9ec4-16555a5142a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:41.335079 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.334414 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs"] Oct 30 23:45:41.340810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.340784 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.348986 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.348932 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6a162557-openshift-default-dockercfg-w6bzl\"" Oct 30 23:45:41.356125 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.354422 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs"] Oct 30 23:45:41.387198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.387153 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-data\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.387376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.387297 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4h2m\" (UniqueName: \"kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-kube-api-access-p4h2m\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.387376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.387350 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-envoy\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.387528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.387454 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-certs\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.387642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.387618 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-podinfo\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.387765 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.387746 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-token\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.387817 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.387800 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-credential-socket\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.387927 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.387904 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e91d97-e0c2-4406-b78a-e4701515b350-istiod-ca-cert\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.387961 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.387952 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-socket\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.483306 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.499634 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-data\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.499736 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4h2m\" (UniqueName: \"kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-kube-api-access-p4h2m\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.499775 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-envoy\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.499814 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-certs\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.499870 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-podinfo\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.499937 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-token\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.499978 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-credential-socket\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.500025 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e91d97-e0c2-4406-b78a-e4701515b350-istiod-ca-cert\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.500058 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-socket\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.506454 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-certs\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.506744 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-data\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.514768 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-socket\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.514892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.514797 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-credential-socket\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.612486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.588187 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e91d97-e0c2-4406-b78a-e4701515b350-istiod-ca-cert\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.638499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.629918 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-podinfo\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.638499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.636220 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-token\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.638499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.637122 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4h2m\" (UniqueName: \"kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-kube-api-access-p4h2m\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.651722 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.650748 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-envoy\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-j2xrs\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.695482 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.691417 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:41.989366 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:45:41.989316 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08111205_a51a_422b_9ec4_16555a5142a7.slice/crio-60f068f24c8e9d5681898d9fceaa35d6e3159103886b9ac34cfee09ec3326b21 WatchSource:0}: Error finding container 60f068f24c8e9d5681898d9fceaa35d6e3159103886b9ac34cfee09ec3326b21: Status 404 returned error can't find the container with id 60f068f24c8e9d5681898d9fceaa35d6e3159103886b9ac34cfee09ec3326b21 Oct 30 23:45:41.989580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:41.989402 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk"] Oct 30 23:45:42.044363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:42.044328 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" event={"ID":"08111205-a51a-422b-9ec4-16555a5142a7","Type":"ContainerStarted","Data":"60f068f24c8e9d5681898d9fceaa35d6e3159103886b9ac34cfee09ec3326b21"} Oct 30 23:45:42.082627 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:45:42.082583 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e91d97_e0c2_4406_b78a_e4701515b350.slice/crio-a83df5f3156facac9cf6c543a5af559e2163833934db553c0dcc7c082b20a158 WatchSource:0}: Error finding container a83df5f3156facac9cf6c543a5af559e2163833934db553c0dcc7c082b20a158: Status 404 returned error can't find the container with id a83df5f3156facac9cf6c543a5af559e2163833934db553c0dcc7c082b20a158 Oct 30 23:45:42.082797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:42.082778 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs"] Oct 30 23:45:42.085574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:42.085537 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:45:42.085688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:42.085602 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:45:42.085688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:42.085633 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:45:43.057136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:43.057093 7570 generic.go:356] "Generic (PLEG): container finished" podID="08111205-a51a-422b-9ec4-16555a5142a7" containerID="6de559f667ef5c2a52063d30974b72ef2887a4c8b1d123df065ef8781c9de399" exitCode=0 Oct 30 23:45:43.057650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:43.057194 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" event={"ID":"08111205-a51a-422b-9ec4-16555a5142a7","Type":"ContainerDied","Data":"6de559f667ef5c2a52063d30974b72ef2887a4c8b1d123df065ef8781c9de399"} Oct 30 23:45:43.059744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:43.059713 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" event={"ID":"b4e91d97-e0c2-4406-b78a-e4701515b350","Type":"ContainerStarted","Data":"3e989089df3c985b7c299b5fd8d0aa156cc68cef18dfd591aaad45c558bd396f"} Oct 30 23:45:43.059855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:43.059764 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" event={"ID":"b4e91d97-e0c2-4406-b78a-e4701515b350","Type":"ContainerStarted","Data":"a83df5f3156facac9cf6c543a5af559e2163833934db553c0dcc7c082b20a158"} Oct 30 23:45:43.105309 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:43.105255 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" podStartSLOduration=2.105241394 podStartE2EDuration="2.105241394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:43.099202262 +0000 UTC m=+1554.618203354" watchObservedRunningTime="2026-10-30 23:45:43.105241394 +0000 UTC m=+1554.624242543" Oct 30 23:45:43.693266 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:43.693234 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:43.698996 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:43.698974 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:44.070350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:44.070266 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:44.071821 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:44.071802 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:45:44.452091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:44.452065 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:44.565304 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:44.565271 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08111205-a51a-422b-9ec4-16555a5142a7-kube-api-access-ttmfn\" (UniqueName: \"kubernetes.io/projected/08111205-a51a-422b-9ec4-16555a5142a7-kube-api-access-ttmfn\") pod \"08111205-a51a-422b-9ec4-16555a5142a7\" (UID: \"08111205-a51a-422b-9ec4-16555a5142a7\") " Oct 30 23:45:44.565487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:44.565401 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-openshift-ca\") pod \"08111205-a51a-422b-9ec4-16555a5142a7\" (UID: \"08111205-a51a-422b-9ec4-16555a5142a7\") " Oct 30 23:45:44.565546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:44.565501 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-xks-ca\") pod \"08111205-a51a-422b-9ec4-16555a5142a7\" (UID: \"08111205-a51a-422b-9ec4-16555a5142a7\") " Oct 30 23:45:44.565758 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:44.565732 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-openshift-ca" pod "08111205-a51a-422b-9ec4-16555a5142a7" (UID: "08111205-a51a-422b-9ec4-16555a5142a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:44.566136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:44.566111 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:44.567745 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:44.567724 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-xks-ca" pod "08111205-a51a-422b-9ec4-16555a5142a7" (UID: "08111205-a51a-422b-9ec4-16555a5142a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:45:44.567859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:44.567818 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08111205-a51a-422b-9ec4-16555a5142a7-kube-api-access-ttmfn" pod "08111205-a51a-422b-9ec4-16555a5142a7" (UID: "08111205-a51a-422b-9ec4-16555a5142a7"). InnerVolumeSpecName "kube-api-access-ttmfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:44.668540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:44.668506 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08111205-a51a-422b-9ec4-16555a5142a7-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:44.668540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:44.668538 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttmfn\" (UniqueName: \"kubernetes.io/projected/08111205-a51a-422b-9ec4-16555a5142a7-kube-api-access-ttmfn\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:45:45.081214 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:45.081180 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" event={"ID":"08111205-a51a-422b-9ec4-16555a5142a7","Type":"ContainerDied","Data":"60f068f24c8e9d5681898d9fceaa35d6e3159103886b9ac34cfee09ec3326b21"} Oct 30 23:45:45.081656 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:45.081237 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60f068f24c8e9d5681898d9fceaa35d6e3159103886b9ac34cfee09ec3326b21" Oct 30 23:45:45.081656 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:45.081190 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk" Oct 30 23:45:46.561218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.561186 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5"] Oct 30 23:45:46.561997 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.561981 7570 cpu_manager.go:414] "containerMap: removing container" podUID="08111205-a51a-422b-9ec4-16555a5142a7" containerName="revoke-keys" Oct 30 23:45:46.562055 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.561999 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08111205-a51a-422b-9ec4-16555a5142a7" containerName="revoke-keys" Oct 30 23:45:46.562168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.562159 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="08111205-a51a-422b-9ec4-16555a5142a7" containerName="revoke-keys" Oct 30 23:45:46.567669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.567647 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.586225 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.585724 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5"] Oct 30 23:45:46.591611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.591298 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-a3e876-serving-cert\"" Oct 30 23:45:46.607269 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.607232 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.607443 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.607363 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqxt\" (UniqueName: \"kubernetes.io/projected/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-kube-api-access-zgqxt\") pod \"maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.607586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.607520 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.607657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.607621 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.709819 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.709771 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.709819 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.709904 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqxt\" (UniqueName: \"kubernetes.io/projected/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-kube-api-access-zgqxt\") pod \"maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.710822 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.710012 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.710822 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.710540 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.711424 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.711152 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.712754 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.712726 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.713654 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.713631 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.722373 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.722352 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqxt\" (UniqueName: \"kubernetes.io/projected/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-kube-api-access-zgqxt\") pod \"maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:46.899816 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:46.899780 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:47.479374 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:45:47.343334 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c5ef3a_fb2d_4f45_9d16_470f553b62c5.slice/crio-ad97c9bf05fe085156a16bd9de7d5c980f08d918cd4ac803e90e7aabb9266386 WatchSource:0}: Error finding container ad97c9bf05fe085156a16bd9de7d5c980f08d918cd4ac803e90e7aabb9266386: Status 404 returned error can't find the container with id ad97c9bf05fe085156a16bd9de7d5c980f08d918cd4ac803e90e7aabb9266386 Oct 30 23:45:47.479374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:47.343884 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5"] Oct 30 23:45:48.130800 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:48.130767 7570 generic.go:356] "Generic (PLEG): container finished" podID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerID="8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738" exitCode=0 Oct 30 23:45:48.131263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:48.130848 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" event={"ID":"48390cfe-87a3-4b4e-9986-8adec60f6349","Type":"ContainerDied","Data":"8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738"} Oct 30 23:45:48.131356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:48.131337 7570 scope.go:122] "RemoveContainer" containerID="8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738" Oct 30 23:45:48.132813 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:48.132794 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" event={"ID":"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5","Type":"ContainerStarted","Data":"8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4"} Oct 30 23:45:48.132916 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:48.132852 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:48.132916 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:48.132880 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" event={"ID":"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5","Type":"ContainerStarted","Data":"ad97c9bf05fe085156a16bd9de7d5c980f08d918cd4ac803e90e7aabb9266386"} Oct 30 23:45:48.133149 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:48.133122 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:45:48.133202 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:48.133189 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:45:48.133236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:48.133223 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:45:48.189046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:48.188992 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podStartSLOduration=1.692417525 podStartE2EDuration="2.188973896s" totalImagesPullingTime="496.556371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:48.1818278 +0000 UTC m=+1559.700828890" watchObservedRunningTime="2026-10-30 23:45:48.188973896 +0000 UTC m=+1559.707974989" Oct 30 23:45:49.144051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:49.144013 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" event={"ID":"48390cfe-87a3-4b4e-9986-8adec60f6349","Type":"ContainerStarted","Data":"82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f"} Oct 30 23:45:49.171755 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:49.171723 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:49.172552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:49.172524 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 23:45:49.172671 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:49.172591 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 23:45:49.814320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:49.814291 7570 scope.go:122] "RemoveContainer" containerID="eae2854ae8e9419f12929a59fa257016f315996d53469ade0938a8acdef2879d" Oct 30 23:45:49.826677 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:49.826654 7570 scope.go:122] "RemoveContainer" containerID="8a244a5fcef2af70eb6c7f0ba666ef14a2e3a63d3c6f09f7a9e3bdbfb24a7d48" Oct 30 23:45:49.837871 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:49.837848 7570 scope.go:122] "RemoveContainer" containerID="6f28d35b39dede69c68bac0f441955dd9ad9bda7fd4ace7b3abe801b272bf5d2" Oct 30 23:45:49.848747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:49.848728 7570 scope.go:122] "RemoveContainer" containerID="29a9817cca5bb5fa13d295de95681b252e9fe6aeed0ec69971c22964829ee84d" Oct 30 23:45:49.859783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:49.859765 7570 scope.go:122] "RemoveContainer" containerID="a3349f1ce336672604ffe9559c69a2151851d01ef5f285d5a2a6a0ae543db807" Oct 30 23:45:49.870224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:49.870197 7570 scope.go:122] "RemoveContainer" containerID="9b1857b3027e8bf6ab0b43fb00ba8bfa7787f6f706adbb5731c02d4bf0f4072d" Oct 30 23:45:50.172089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:50.172058 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 23:45:50.172517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:50.172117 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 23:45:51.026731 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:51.026703 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:45:51.182538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:51.177455 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 23:45:51.182538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:51.177532 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 23:45:52.171387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:52.171352 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 23:45:52.171580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:52.171409 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 23:45:53.171873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:53.171835 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 23:45:53.172327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:53.171894 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 23:45:54.149663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:54.149636 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:45:54.171842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:54.171812 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 23:45:54.171990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:54.171865 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 23:45:55.152801 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:55.152765 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:55.178763 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:55.178733 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:55.252043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:55.252005 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:45:59.502493 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:45:59.498698 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:45:59.555364 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:59.555315 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk"] Oct 30 23:45:59.559817 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:45:59.559792 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4285d4a-9hpjk"] Oct 30 23:46:00.614835 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:00.614796 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08111205-a51a-422b-9ec4-16555a5142a7" path="/var/lib/kubelet/pods/08111205-a51a-422b-9ec4-16555a5142a7/volumes" Oct 30 23:46:12.032045 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:12.032012 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9"] Oct 30 23:46:12.036310 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:12.036286 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5cb38a-7v9w9"] Oct 30 23:46:12.606995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:12.606959 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa11a2b2-7ab3-405c-bd19-01499d1867ed" path="/var/lib/kubelet/pods/aa11a2b2-7ab3-405c-bd19-01499d1867ed/volumes" Oct 30 23:46:19.038345 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:19.038304 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294"] Oct 30 23:46:19.043496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:19.043443 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a72cba9-m6294"] Oct 30 23:46:20.604969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:20.604933 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b" path="/var/lib/kubelet/pods/9de3ab3b-0dca-43b2-b401-ba8f35dc5d3b/volumes" Oct 30 23:46:23.050491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:23.049514 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp"] Oct 30 23:46:23.077498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:23.076037 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2373f5-8l4hp"] Oct 30 23:46:24.607335 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:24.607286 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c" path="/var/lib/kubelet/pods/e79b52a7-5ecd-40f1-a52f-bd5f82fb2f8c/volumes" Oct 30 23:46:29.416707 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:46:29.416661 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:46:42.044406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:42.044370 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq"] Oct 30 23:46:42.048669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:42.048640 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cd6a0d-t4mrq"] Oct 30 23:46:42.605413 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:42.605378 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440b4040-ef5a-41e3-a505-7ca11b054f05" path="/var/lib/kubelet/pods/440b4040-ef5a-41e3-a505-7ca11b054f05/volumes" Oct 30 23:46:44.024762 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:44.024723 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v"] Oct 30 23:46:44.028838 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:44.028809 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1572848f-bbx4v"] Oct 30 23:46:44.605602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:44.605572 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca3a700-3162-4b5a-861f-69b6bee3d3e9" path="/var/lib/kubelet/pods/7ca3a700-3162-4b5a-861f-69b6bee3d3e9/volumes" Oct 30 23:46:46.772361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.772324 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8"] Oct 30 23:46:46.778819 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.778794 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:46.801851 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.801742 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8"] Oct 30 23:46:46.852170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.852135 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b6cb71-openshift-default-dockercfg-jl9st\"" Oct 30 23:46:46.905486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.904642 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-envoy\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:46.905486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.904695 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-data\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:46.905486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.904771 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-socket\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:46.905486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.904818 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xcdx\" (UniqueName: \"kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-kube-api-access-2xcdx\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:46.905486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.904896 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-credential-socket\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:46.905486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.905077 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-istio-token\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:46.905486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.905106 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-certs\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:46.905486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.905308 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71246733-216c-48db-8261-cef843d9b6b4-istio-podinfo\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:46.905486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:46.905402 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71246733-216c-48db-8261-cef843d9b6b4-istiod-ca-cert\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.014910 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.014864 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-certs\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.015832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.015808 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71246733-216c-48db-8261-cef843d9b6b4-istio-podinfo\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.016550 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.016530 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71246733-216c-48db-8261-cef843d9b6b4-istiod-ca-cert\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.017454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.017434 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-envoy\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.033684 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.033614 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-data\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.033905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.033886 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-socket\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.034057 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.034041 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xcdx\" (UniqueName: \"kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-kube-api-access-2xcdx\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.034237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.034220 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-credential-socket\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.054490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.034446 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-socket\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.054490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.017330 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71246733-216c-48db-8261-cef843d9b6b4-istiod-ca-cert\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.054490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.015593 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-certs\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.054490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.035033 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-credential-socket\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.054490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.034230 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-data\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.054490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.037860 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-envoy\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.054490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.044500 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71246733-216c-48db-8261-cef843d9b6b4-istio-podinfo\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.055499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.055451 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-istio-token\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.081489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.080650 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xcdx\" (UniqueName: \"kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-kube-api-access-2xcdx\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.093483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.090131 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-istio-token\") pod \"e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.099525 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.099497 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:47.490223 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.490189 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8"] Oct 30 23:46:47.490395 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:46:47.490374 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71246733_216c_48db_8261_cef843d9b6b4.slice/crio-c17f60e4dbe08e11e7e6d1610838c085e3c27b01a9bbed903e8857389eecf85c WatchSource:0}: Error finding container c17f60e4dbe08e11e7e6d1610838c085e3c27b01a9bbed903e8857389eecf85c: Status 404 returned error can't find the container with id c17f60e4dbe08e11e7e6d1610838c085e3c27b01a9bbed903e8857389eecf85c Oct 30 23:46:47.494080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.494040 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:46:47.494187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.494138 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:46:47.494187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:47.494181 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:46:48.025843 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:48.025805 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb"] Oct 30 23:46:48.030242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:48.030213 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0074fabf-xc9fb"] Oct 30 23:46:48.225354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:48.225321 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" event={"ID":"71246733-216c-48db-8261-cef843d9b6b4","Type":"ContainerStarted","Data":"18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77"} Oct 30 23:46:48.225575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:48.225376 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" event={"ID":"71246733-216c-48db-8261-cef843d9b6b4","Type":"ContainerStarted","Data":"c17f60e4dbe08e11e7e6d1610838c085e3c27b01a9bbed903e8857389eecf85c"} Oct 30 23:46:48.251975 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:48.251922 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" podStartSLOduration=2.251908929 podStartE2EDuration="2.251908929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:48.247220483 +0000 UTC m=+1619.766221576" watchObservedRunningTime="2026-10-30 23:46:48.251908929 +0000 UTC m=+1619.770910018" Oct 30 23:46:48.616043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:48.616009 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85070cda-919f-4fd5-ad70-d50b6852248c" path="/var/lib/kubelet/pods/85070cda-919f-4fd5-ad70-d50b6852248c/volumes" Oct 30 23:46:49.104881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:49.104830 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:49.127173 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:49.127129 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:49.234267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:49.234231 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:49.235146 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:49.235130 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:46:50.003119 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:50.003083 7570 scope.go:122] "RemoveContainer" containerID="19d696a30b62f93a455339f235b5c0512a5bd9394ab6ef21cf2a4662b94f0d3d" Oct 30 23:46:50.018237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:50.018211 7570 scope.go:122] "RemoveContainer" containerID="63002016fc0ff19546e215b9c75e58734088649be676d5e13403a6adb62f5579" Oct 30 23:46:50.031715 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:50.031692 7570 scope.go:122] "RemoveContainer" containerID="4b95e5f209f3cbf2a416c6d7059d556e05005bbf01796d10a7d0b5652c673a34" Oct 30 23:46:50.042881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:50.042855 7570 scope.go:122] "RemoveContainer" containerID="8bd49990dcb203d479e1d86ce65d3b28b1b7edd83ef9d1d0fe8d17d0cb4c1ace" Oct 30 23:46:50.054476 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:50.054444 7570 scope.go:122] "RemoveContainer" containerID="3f1e5e7b3ed99157e38247533443a20f4136463a9ca50dcb3d8a75b4d7e16f3f" Oct 30 23:46:50.066643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:50.066620 7570 scope.go:122] "RemoveContainer" containerID="4ae427a6d9f9ea44211f4ddbb96cc383f19e4cc46d8c01b8dc3fc8ba35296f0e" Oct 30 23:46:55.026065 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:55.026032 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd"] Oct 30 23:46:55.032335 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:55.032305 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8889cf6f-tngpd"] Oct 30 23:46:56.605136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:56.605103 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2686bb47-640a-4dc2-81fb-945e17f98906" path="/var/lib/kubelet/pods/2686bb47-640a-4dc2-81fb-945e17f98906/volumes" Oct 30 23:46:59.413746 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:46:59.413653 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:46:59.452066 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.452034 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c"] Oct 30 23:46:59.459104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.459076 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:46:59.469244 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.469214 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b6cb71-serving-cert\"" Oct 30 23:46:59.469867 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.469840 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c"] Oct 30 23:46:59.604056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.603973 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-openshift-service-ca\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:46:59.604410 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.604380 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-maas-api-tls\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:46:59.604581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.604418 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-metrics-certs\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:46:59.604654 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.604584 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfpj8\" (UniqueName: \"kubernetes.io/projected/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-kube-api-access-lfpj8\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:46:59.706677 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.706563 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfpj8\" (UniqueName: \"kubernetes.io/projected/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-kube-api-access-lfpj8\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:46:59.706927 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.706909 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-openshift-service-ca\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:46:59.707039 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.707021 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-maas-api-tls\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:46:59.707095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.707053 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-metrics-certs\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:46:59.707194 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:46:59.707169 7570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-serving-cert: secret "maas-api-e2e-api-b-b6cb71-serving-cert" not found Oct 30 23:46:59.707353 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:46:59.707280 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-maas-api-tls podName:6cd6943f-aabc-4990-be83-d7d6a0b8fa38 nodeName:}" failed. No retries permitted until 2026-10-30 23:47:00.207257975 +0000 UTC m=+1631.726259043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-maas-api-tls") pod "maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" (UID: "6cd6943f-aabc-4990-be83-d7d6a0b8fa38") : secret "maas-api-e2e-api-b-b6cb71-serving-cert" not found Oct 30 23:46:59.707683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.707665 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-openshift-service-ca\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:46:59.710044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.710020 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-metrics-certs\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:46:59.719832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:46:59.719806 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfpj8\" (UniqueName: \"kubernetes.io/projected/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-kube-api-access-lfpj8\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:47:00.220114 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:00.220079 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-maas-api-tls\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:47:00.223086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:00.223056 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-maas-api-tls\") pod \"maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:47:00.384208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:00.384177 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:47:00.672321 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:00.672292 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c"] Oct 30 23:47:00.672688 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:47:00.672290 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd6943f_aabc_4990_be83_d7d6a0b8fa38.slice/crio-cc7a80534542205348bebe65986116f1199497e6007d50161056ce7229b4839f WatchSource:0}: Error finding container cc7a80534542205348bebe65986116f1199497e6007d50161056ce7229b4839f: Status 404 returned error can't find the container with id cc7a80534542205348bebe65986116f1199497e6007d50161056ce7229b4839f Oct 30 23:47:01.340362 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:01.340326 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" event={"ID":"6cd6943f-aabc-4990-be83-d7d6a0b8fa38","Type":"ContainerStarted","Data":"d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d"} Oct 30 23:47:01.340533 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:01.340390 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:47:01.340533 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:01.340407 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" event={"ID":"6cd6943f-aabc-4990-be83-d7d6a0b8fa38","Type":"ContainerStarted","Data":"cc7a80534542205348bebe65986116f1199497e6007d50161056ce7229b4839f"} Oct 30 23:47:01.364919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:01.364865 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" podStartSLOduration=1.799135615 podStartE2EDuration="2.364851005s" totalImagesPullingTime="565.71539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:01.360306804 +0000 UTC m=+1632.879307892" watchObservedRunningTime="2026-10-30 23:47:01.364851005 +0000 UTC m=+1632.883852120" Oct 30 23:47:02.025810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:02.025774 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6"] Oct 30 23:47:02.030038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:02.030010 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441f67c6-xfmh6"] Oct 30 23:47:02.605028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:02.604994 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd65f927-691b-412a-9dcb-2b79fca2213e" path="/var/lib/kubelet/pods/fd65f927-691b-412a-9dcb-2b79fca2213e/volumes" Oct 30 23:47:07.390488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:07.389646 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:47:07.649553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:07.648874 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 30 23:47:07.744830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:07.743582 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:07.781621 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:07.781554 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:07.918832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:07.898547 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": context deadline exceeded" Oct 30 23:47:08.055598 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:08.040250 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:08.055598 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:08.052952 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:08.069845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:08.068097 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:08.069845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:08.069531 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:08.181843 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:08.181445 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:08.311488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:08.300875 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:47:08.371480 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:08.368724 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:08.408215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:08.408170 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:08.454645 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:08.454541 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:08.750659 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:08.750549 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:08.752011 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:08.751986 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:47:08.864580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:08.860317 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:47:15.028419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:15.028375 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97"] Oct 30 23:47:15.033990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:15.033962 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192be83-fnp97"] Oct 30 23:47:15.225315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:15.225281 7570 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:47:15.225615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:15.225350 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:16.604650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:16.604618 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbc604c-0936-4e1b-87c8-66d7e8397337" path="/var/lib/kubelet/pods/1dbc604c-0936-4e1b-87c8-66d7e8397337/volumes" Oct 30 23:47:23.023897 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:23.023859 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr"] Oct 30 23:47:23.028361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:23.028328 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c82df7b-cnbsr"] Oct 30 23:47:24.605139 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:24.605103 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefad00c-f5e3-490c-a477-8a7f643ba668" path="/var/lib/kubelet/pods/fefad00c-f5e3-490c-a477-8a7f643ba668/volumes" Oct 30 23:47:25.904132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:25.904098 7570 generic.go:356] "Generic (PLEG): container finished" podID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerID="3e989089df3c985b7c299b5fd8d0aa156cc68cef18dfd591aaad45c558bd396f" exitCode=0 Oct 30 23:47:25.904648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:25.904186 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" event={"ID":"b4e91d97-e0c2-4406-b78a-e4701515b350","Type":"ContainerDied","Data":"3e989089df3c985b7c299b5fd8d0aa156cc68cef18dfd591aaad45c558bd396f"} Oct 30 23:47:25.904715 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:25.904701 7570 scope.go:122] "RemoveContainer" containerID="3e989089df3c985b7c299b5fd8d0aa156cc68cef18dfd591aaad45c558bd396f" Oct 30 23:47:25.906587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:25.906559 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:47:25.906673 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:25.906629 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:47:25.906673 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:25.906660 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:47:26.936252 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:26.936213 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" event={"ID":"b4e91d97-e0c2-4406-b78a-e4701515b350","Type":"ContainerStarted","Data":"d8b2cd330f19776fc3ffc3dadda62011a670aae92fc5df4f7240fcfd3c3777f4"} Oct 30 23:47:27.704606 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:27.704576 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:47:27.705054 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:27.705031 7570 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 23:47:27.705251 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:27.705226 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 23:47:28.024212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:28.024125 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2"] Oct 30 23:47:28.028175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:28.028154 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5106cbe-bj7p2"] Oct 30 23:47:28.624451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:28.624417 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e518bd-b5ec-4504-85fa-8c012807bbe0" path="/var/lib/kubelet/pods/85e518bd-b5ec-4504-85fa-8c012807bbe0/volumes" Oct 30 23:47:28.696960 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:28.696922 7570 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 23:47:28.697253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:28.697222 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 23:47:29.070209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:29.070133 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:47:29.462161 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:47:29.462124 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:47:29.696662 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:29.694836 7570 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 23:47:29.696662 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:29.694901 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 23:47:30.708677 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:30.708640 7570 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 23:47:30.709296 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:30.709261 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 23:47:31.044544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:31.044511 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r"] Oct 30 23:47:31.048379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:31.048360 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76e6f62-rdp9r"] Oct 30 23:47:31.715877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:31.715838 7570 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 23:47:31.716415 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:31.715916 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 23:47:32.605383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:32.605349 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520a1cb3-77dc-47d6-add9-0c262b514835" path="/var/lib/kubelet/pods/520a1cb3-77dc-47d6-add9-0c262b514835/volumes" Oct 30 23:47:32.697512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:32.697443 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:47:32.698668 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:32.698647 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:47:35.024667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:35.024631 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh"] Oct 30 23:47:35.028494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:35.028454 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f5daa2f-9t8qh"] Oct 30 23:47:36.604653 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:36.604617 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086ece5e-33b4-46d3-bef7-48e28cc5ad3e" path="/var/lib/kubelet/pods/086ece5e-33b4-46d3-bef7-48e28cc5ad3e/volumes" Oct 30 23:47:46.035717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:46.035678 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k"] Oct 30 23:47:46.039049 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:46.039023 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c880c983-g884k"] Oct 30 23:47:46.645120 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:46.644650 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0" path="/var/lib/kubelet/pods/9f8bc5a9-5d04-46fc-bbcb-ba005613f2e0/volumes" Oct 30 23:47:50.138110 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:50.138080 7570 scope.go:122] "RemoveContainer" containerID="7c2e7bf7ac10359f0ff0fcd9dc24c198906da34288dc320d68ed9febcf54433e" Oct 30 23:47:50.156116 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:50.156081 7570 scope.go:122] "RemoveContainer" containerID="1fc90eaf4f6f93b847311fc784f05e7da646fef533cf66e4c37ac13777a82cda" Oct 30 23:47:50.168694 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:50.168669 7570 scope.go:122] "RemoveContainer" containerID="0bf02f0a1708240c4b1d5393e05de936d4cc65b7ebf4f642cb056bdab31fe987" Oct 30 23:47:50.182105 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:50.182084 7570 scope.go:122] "RemoveContainer" containerID="cb8d31452d461f987a167ab59413f5ab2deac9f0cb4c1b8020a9c67c2ad25258" Oct 30 23:47:50.199051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:50.199025 7570 scope.go:122] "RemoveContainer" containerID="eb0dba80b3ea9d8ef87bc91ff93a6a9814ba07075c2fc2be805ec37e5ff0a4e8" Oct 30 23:47:50.214396 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:50.214372 7570 scope.go:122] "RemoveContainer" containerID="ccfaa9afd03f2b3057283ed590fb9cedcbdf99f62068007c88a5f703be073dbe" Oct 30 23:47:50.276608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:50.276570 7570 patch_prober.go:29] interesting pod/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:47:50.276762 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:50.276636 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:47:52.027338 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:52.027282 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd"] Oct 30 23:47:52.033863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:52.033834 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b1f042-dpmmd"] Oct 30 23:47:52.604376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:52.604336 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57b9191-7694-4e32-acce-b6275d03180b" path="/var/lib/kubelet/pods/d57b9191-7694-4e32-acce-b6275d03180b/volumes" Oct 30 23:47:57.026861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:57.026824 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv"] Oct 30 23:47:57.032675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:57.032647 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522c69d8-562nv"] Oct 30 23:47:58.605803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:58.605772 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea62cc2-1531-40fd-a104-9dcfb2d9e72f" path="/var/lib/kubelet/pods/aea62cc2-1531-40fd-a104-9dcfb2d9e72f/volumes" Oct 30 23:47:59.109503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:59.105200 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq"] Oct 30 23:47:59.837063 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:47:59.837023 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:47:59.854129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:59.854093 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq"] Oct 30 23:47:59.887232 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:59.887197 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 30 23:47:59.978284 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:59.978249 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-200408\"" Oct 30 23:47:59.978575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:47:59.978557 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-200408-dockercfg-zpwrj\"" Oct 30 23:48:00.024432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:00.024399 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65426\" (UniqueName: \"kubernetes.io/projected/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-kube-api-access-65426\") pod \"payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq\" (UID: \"201fbcb8-8fed-44e3-b60d-4b89d8c114ca\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 30 23:48:00.024759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:00.024735 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq\" (UID: \"201fbcb8-8fed-44e3-b60d-4b89d8c114ca\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 30 23:48:00.126376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:00.126290 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65426\" (UniqueName: \"kubernetes.io/projected/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-kube-api-access-65426\") pod \"payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq\" (UID: \"201fbcb8-8fed-44e3-b60d-4b89d8c114ca\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 30 23:48:00.126592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:00.126571 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq\" (UID: \"201fbcb8-8fed-44e3-b60d-4b89d8c114ca\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 30 23:48:00.127256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:00.127237 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq\" (UID: \"201fbcb8-8fed-44e3-b60d-4b89d8c114ca\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 30 23:48:00.143232 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:00.143202 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65426\" (UniqueName: \"kubernetes.io/projected/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-kube-api-access-65426\") pod \"payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq\" (UID: \"201fbcb8-8fed-44e3-b60d-4b89d8c114ca\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 30 23:48:00.322583 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:00.322547 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 30 23:48:01.031034 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:01.030953 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9"] Oct 30 23:48:01.037057 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:01.037028 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db63510-k62l9"] Oct 30 23:48:01.711934 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:01.711905 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq"] Oct 30 23:48:01.713276 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:48:01.713241 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201fbcb8_8fed_44e3_b60d_4b89d8c114ca.slice/crio-5c82ff100bdb042a1b38dd79422ee4ac0e1b8885202939592b84bb7240f60de5 WatchSource:0}: Error finding container 5c82ff100bdb042a1b38dd79422ee4ac0e1b8885202939592b84bb7240f60de5: Status 404 returned error can't find the container with id 5c82ff100bdb042a1b38dd79422ee4ac0e1b8885202939592b84bb7240f60de5 Oct 30 23:48:02.576113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:02.576070 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" event={"ID":"201fbcb8-8fed-44e3-b60d-4b89d8c114ca","Type":"ContainerStarted","Data":"a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f"} Oct 30 23:48:02.576652 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:02.576140 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" event={"ID":"201fbcb8-8fed-44e3-b60d-4b89d8c114ca","Type":"ContainerStarted","Data":"5c82ff100bdb042a1b38dd79422ee4ac0e1b8885202939592b84bb7240f60de5"} Oct 30 23:48:02.576652 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:02.576172 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 30 23:48:02.607852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:02.607804 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" podStartSLOduration=4.607790623 podStartE2EDuration="4.607790623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:02.602927647 +0000 UTC m=+1694.121928739" watchObservedRunningTime="2026-10-30 23:48:02.607790623 +0000 UTC m=+1694.126791713" Oct 30 23:48:02.609329 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:02.609304 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3687f3e8-fe18-460b-b448-95c82af92975" path="/var/lib/kubelet/pods/3687f3e8-fe18-460b-b448-95c82af92975/volumes" Oct 30 23:48:04.878521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:04.878448 7570 patch_prober.go:29] interesting pod/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": read tcp 10.134.0.2:55102->10.134.0.136:15021: read: connection reset by peer" start-of-body= Oct 30 23:48:04.879689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:04.879658 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": read tcp 10.134.0.2:55102->10.134.0.136:15021: read: connection reset by peer" Oct 30 23:48:05.652097 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:05.652067 7570 generic.go:356] "Generic (PLEG): container finished" podID="71246733-216c-48db-8261-cef843d9b6b4" containerID="18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77" exitCode=0 Oct 30 23:48:05.652301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:05.652142 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" event={"ID":"71246733-216c-48db-8261-cef843d9b6b4","Type":"ContainerDied","Data":"18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77"} Oct 30 23:48:05.652655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:05.652638 7570 scope.go:122] "RemoveContainer" containerID="18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77" Oct 30 23:48:05.658891 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:05.658860 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:48:05.659023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:05.658946 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:48:05.659023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:05.658983 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:48:06.735488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:06.735262 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" event={"ID":"71246733-216c-48db-8261-cef843d9b6b4","Type":"ContainerStarted","Data":"0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6"} Oct 30 23:48:07.104491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:07.102898 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:48:07.104491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:07.103034 7570 patch_prober.go:29] interesting pod/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 23:48:07.104491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:07.103080 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 23:48:08.123624 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:08.123594 7570 patch_prober.go:29] interesting pod/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 23:48:08.124161 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:08.123642 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 23:48:09.109489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:09.102861 7570 patch_prober.go:29] interesting pod/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 23:48:09.109489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:09.102922 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 23:48:10.037778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:10.036657 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz"] Oct 30 23:48:10.042507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:10.042434 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f795a373-lxvrz"] Oct 30 23:48:10.112385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:10.111948 7570 patch_prober.go:29] interesting pod/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 23:48:10.112385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:10.112020 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 23:48:10.621305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:10.621272 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac9cff3-e332-4a93-98a4-7ba14bd679f3" path="/var/lib/kubelet/pods/dac9cff3-e332-4a93-98a4-7ba14bd679f3/volumes" Oct 30 23:48:11.102825 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:11.102789 7570 patch_prober.go:29] interesting pod/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 23:48:11.103432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:11.103401 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 23:48:12.108487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:12.105837 7570 patch_prober.go:29] interesting pod/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 23:48:12.108487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:12.105910 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 23:48:13.103790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:13.103758 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:48:13.104068 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:13.104050 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:48:13.104711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:13.104689 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:48:13.638040 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:13.638013 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 30 23:48:24.024389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:24.024349 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r"] Oct 30 23:48:24.027977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:24.027949 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efba4c9-fkh9r"] Oct 30 23:48:24.604418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:24.604383 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d6813e-e082-4555-a067-2a7b062a60ad" path="/var/lib/kubelet/pods/99d6813e-e082-4555-a067-2a7b062a60ad/volumes" Oct 30 23:48:29.418905 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:48:29.418804 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:48:34.874508 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:34.870328 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4"] Oct 30 23:48:34.877437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:34.876403 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 30 23:48:34.887860 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:34.887835 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4"] Oct 30 23:48:34.909966 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:34.909058 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-86d846-dockercfg-dc664\"" Oct 30 23:48:34.950029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:34.950001 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-86d846\"" Oct 30 23:48:35.147444 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.138704 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6pk\" (UniqueName: \"kubernetes.io/projected/ca8613ba-3272-43d3-9661-33618f859cff-kube-api-access-dq6pk\") pod \"payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4\" (UID: \"ca8613ba-3272-43d3-9661-33618f859cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 30 23:48:35.147444 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.138851 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8613ba-3272-43d3-9661-33618f859cff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4\" (UID: \"ca8613ba-3272-43d3-9661-33618f859cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 30 23:48:35.249484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.242147 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6pk\" (UniqueName: \"kubernetes.io/projected/ca8613ba-3272-43d3-9661-33618f859cff-kube-api-access-dq6pk\") pod \"payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4\" (UID: \"ca8613ba-3272-43d3-9661-33618f859cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 30 23:48:35.249484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.242700 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8613ba-3272-43d3-9661-33618f859cff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4\" (UID: \"ca8613ba-3272-43d3-9661-33618f859cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 30 23:48:35.249484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.243595 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8613ba-3272-43d3-9661-33618f859cff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4\" (UID: \"ca8613ba-3272-43d3-9661-33618f859cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 30 23:48:35.264418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.264374 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6pk\" (UniqueName: \"kubernetes.io/projected/ca8613ba-3272-43d3-9661-33618f859cff-kube-api-access-dq6pk\") pod \"payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4\" (UID: \"ca8613ba-3272-43d3-9661-33618f859cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 30 23:48:35.499194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.499094 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 30 23:48:35.806297 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.806228 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4"] Oct 30 23:48:35.820061 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.820031 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk"] Oct 30 23:48:35.831154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.831120 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.845784 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.845757 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1bc837-openshift-default-dockercfg-5htq6\"" Oct 30 23:48:35.850947 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.850904 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk"] Oct 30 23:48:35.865958 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.865926 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-envoy\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.866195 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.865972 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-data\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.866195 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.866000 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-token\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.866195 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.866123 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-credential-socket\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.866976 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.866353 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42jwx\" (UniqueName: \"kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-kube-api-access-42jwx\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.866976 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.866444 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istiod-ca-cert\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.866976 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.866618 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-socket\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.866976 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.866662 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-certs\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.866976 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.866745 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-podinfo\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.968977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.968939 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-envoy\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.969486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.968990 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-data\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.969486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.969018 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-token\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.969486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.969047 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-credential-socket\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.969486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.969243 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42jwx\" (UniqueName: \"kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-kube-api-access-42jwx\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.969486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.969311 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istiod-ca-cert\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.969486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.969395 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-data\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.969904 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.969505 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-credential-socket\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.969904 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.969576 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-socket\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.969904 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.969629 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-certs\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.969904 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.969712 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-podinfo\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.969904 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.969878 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-socket\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.970421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.970061 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-certs\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.970421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.970174 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istiod-ca-cert\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.972222 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.972197 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-envoy\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.972578 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.972559 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-podinfo\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.983535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.983507 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-token\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:35.985211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:35.985183 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42jwx\" (UniqueName: \"kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-kube-api-access-42jwx\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:36.159142 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:36.159106 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:36.618094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:36.618056 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" event={"ID":"ca8613ba-3272-43d3-9661-33618f859cff","Type":"ContainerStarted","Data":"4f678db60e39015edd8c36718ebafbe5657d2770f28741621f2345eeaa93dc40"} Oct 30 23:48:36.618339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:36.618113 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" event={"ID":"ca8613ba-3272-43d3-9661-33618f859cff","Type":"ContainerStarted","Data":"659eac5f22524fc511fe71b153e5eb46fe25bf6c1e3d6abc41dcecd917bcdfd0"} Oct 30 23:48:36.618339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:36.618136 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 30 23:48:36.629044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:36.628990 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" podStartSLOduration=2.628975339 podStartE2EDuration="2.628975339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:36.62436087 +0000 UTC m=+1728.143361959" watchObservedRunningTime="2026-10-30 23:48:36.628975339 +0000 UTC m=+1728.147976430" Oct 30 23:48:36.671401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:36.671374 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk"] Oct 30 23:48:36.672234 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:48:36.672205 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51164b20_de0a_4b55_bdb7_7c25c2d5d922.slice/crio-73096d00f86e882a9154270a63f58b9ec11fd430d360328e89faa87b8232d85c WatchSource:0}: Error finding container 73096d00f86e882a9154270a63f58b9ec11fd430d360328e89faa87b8232d85c: Status 404 returned error can't find the container with id 73096d00f86e882a9154270a63f58b9ec11fd430d360328e89faa87b8232d85c Oct 30 23:48:36.675132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:36.675098 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:48:36.675237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:36.675167 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:48:36.675237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:36.675198 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:48:37.616555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:37.616519 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" event={"ID":"51164b20-de0a-4b55-bdb7-7c25c2d5d922","Type":"ContainerStarted","Data":"982b2a7324033b2bc083f03b651ac5a85ec2d52e6b10437d961438545835dea8"} Oct 30 23:48:37.617029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:37.616580 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" event={"ID":"51164b20-de0a-4b55-bdb7-7c25c2d5d922","Type":"ContainerStarted","Data":"73096d00f86e882a9154270a63f58b9ec11fd430d360328e89faa87b8232d85c"} Oct 30 23:48:38.159786 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.159753 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:38.165253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.165227 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:38.217987 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.217937 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podStartSLOduration=3.217922211 podStartE2EDuration="3.217922211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:37.649730026 +0000 UTC m=+1729.168731118" watchObservedRunningTime="2026-10-30 23:48:38.217922211 +0000 UTC m=+1729.736923377" Oct 30 23:48:38.626851 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.626822 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:38.628189 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.628163 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:48:38.630523 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.630500 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2"] Oct 30 23:48:38.635322 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.635297 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 30 23:48:38.661635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.661612 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2"] Oct 30 23:48:38.665983 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.665957 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131867e6-7957-48d7-96d5-50f9e8ff45b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2\" (UID: \"131867e6-7957-48d7-96d5-50f9e8ff45b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 30 23:48:38.666181 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.666155 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkjq\" (UniqueName: \"kubernetes.io/projected/131867e6-7957-48d7-96d5-50f9e8ff45b2-kube-api-access-6nkjq\") pod \"payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2\" (UID: \"131867e6-7957-48d7-96d5-50f9e8ff45b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 30 23:48:38.670616 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.670592 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-885999-dockercfg-tmfpm\"" Oct 30 23:48:38.677679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.677652 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-885999\"" Oct 30 23:48:38.826488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.824854 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131867e6-7957-48d7-96d5-50f9e8ff45b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2\" (UID: \"131867e6-7957-48d7-96d5-50f9e8ff45b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 30 23:48:38.826488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.824946 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkjq\" (UniqueName: \"kubernetes.io/projected/131867e6-7957-48d7-96d5-50f9e8ff45b2-kube-api-access-6nkjq\") pod \"payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2\" (UID: \"131867e6-7957-48d7-96d5-50f9e8ff45b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 30 23:48:38.826488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:38.826371 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131867e6-7957-48d7-96d5-50f9e8ff45b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2\" (UID: \"131867e6-7957-48d7-96d5-50f9e8ff45b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 30 23:48:39.023913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:39.023837 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkjq\" (UniqueName: \"kubernetes.io/projected/131867e6-7957-48d7-96d5-50f9e8ff45b2-kube-api-access-6nkjq\") pod \"payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2\" (UID: \"131867e6-7957-48d7-96d5-50f9e8ff45b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 30 23:48:39.251500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:39.250923 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 30 23:48:39.605990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:39.605965 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2"] Oct 30 23:48:39.606511 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:48:39.606479 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131867e6_7957_48d7_96d5_50f9e8ff45b2.slice/crio-154cd04b7632922f6fa863643d7d35c2de9327718fbbeb3e91976dd01b49bae0 WatchSource:0}: Error finding container 154cd04b7632922f6fa863643d7d35c2de9327718fbbeb3e91976dd01b49bae0: Status 404 returned error can't find the container with id 154cd04b7632922f6fa863643d7d35c2de9327718fbbeb3e91976dd01b49bae0 Oct 30 23:48:39.641127 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:39.641090 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" event={"ID":"131867e6-7957-48d7-96d5-50f9e8ff45b2","Type":"ContainerStarted","Data":"154cd04b7632922f6fa863643d7d35c2de9327718fbbeb3e91976dd01b49bae0"} Oct 30 23:48:40.030246 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.030154 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c"] Oct 30 23:48:40.030418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.030383 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" podUID="6cd6943f-aabc-4990-be83-d7d6a0b8fa38" containerName="maas-api" containerID="cri-o://d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d" gracePeriod=30 Oct 30 23:48:40.362129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.362099 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:48:40.449487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.449132 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-metrics-certs\") pod \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " Oct 30 23:48:40.449487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.449223 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-kube-api-access-lfpj8\" (UniqueName: \"kubernetes.io/projected/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-kube-api-access-lfpj8\") pod \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " Oct 30 23:48:40.449487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.449312 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-maas-api-tls\") pod \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " Oct 30 23:48:40.449825 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.449527 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-openshift-service-ca\") pod \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\" (UID: \"6cd6943f-aabc-4990-be83-d7d6a0b8fa38\") " Oct 30 23:48:40.472497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.465704 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-openshift-service-ca" pod "6cd6943f-aabc-4990-be83-d7d6a0b8fa38" (UID: "6cd6943f-aabc-4990-be83-d7d6a0b8fa38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:48:40.512488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.503933 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-maas-api-tls" pod "6cd6943f-aabc-4990-be83-d7d6a0b8fa38" (UID: "6cd6943f-aabc-4990-be83-d7d6a0b8fa38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:48:40.576487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.571701 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:40.576487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.571743 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:40.679707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.672540 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-kube-api-access-lfpj8" pod "6cd6943f-aabc-4990-be83-d7d6a0b8fa38" (UID: "6cd6943f-aabc-4990-be83-d7d6a0b8fa38"). InnerVolumeSpecName "kube-api-access-lfpj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:48:40.679707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.674353 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfpj8\" (UniqueName: \"kubernetes.io/projected/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-kube-api-access-lfpj8\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:40.679707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.678522 7570 generic.go:356] "Generic (PLEG): container finished" podID="6cd6943f-aabc-4990-be83-d7d6a0b8fa38" containerID="d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d" exitCode=0 Oct 30 23:48:40.679707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.678833 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" Oct 30 23:48:40.694852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.689872 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-metrics-certs" pod "6cd6943f-aabc-4990-be83-d7d6a0b8fa38" (UID: "6cd6943f-aabc-4990-be83-d7d6a0b8fa38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:48:40.694852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.692777 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" event={"ID":"6cd6943f-aabc-4990-be83-d7d6a0b8fa38","Type":"ContainerDied","Data":"d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d"} Oct 30 23:48:40.694852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.692836 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c" event={"ID":"6cd6943f-aabc-4990-be83-d7d6a0b8fa38","Type":"ContainerDied","Data":"cc7a80534542205348bebe65986116f1199497e6007d50161056ce7229b4839f"} Oct 30 23:48:40.694852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.692864 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 30 23:48:40.694852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.692890 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" event={"ID":"131867e6-7957-48d7-96d5-50f9e8ff45b2","Type":"ContainerStarted","Data":"3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4"} Oct 30 23:48:40.694852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.693112 7570 scope.go:122] "RemoveContainer" containerID="d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d" Oct 30 23:48:40.718846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.715853 7570 scope.go:122] "RemoveContainer" containerID="d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d" Oct 30 23:48:40.718846 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:48:40.716227 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d\": container with ID starting with d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d not found: ID does not exist" containerID="d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d" Oct 30 23:48:40.718846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.716261 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d"} err="failed to get container status \"d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d\": rpc error: code = NotFound desc = could not find container \"d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d\": container with ID starting with d12d5f2eb62f56b6e794380c31284170cadea087c5296c7b318e6325a929657d not found: ID does not exist" Oct 30 23:48:40.718846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.716923 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" podStartSLOduration=2.716905059 podStartE2EDuration="2.716905059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:40.710008444 +0000 UTC m=+1732.229009536" watchObservedRunningTime="2026-10-30 23:48:40.716905059 +0000 UTC m=+1732.235906153" Oct 30 23:48:40.776873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:40.776846 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cd6943f-aabc-4990-be83-d7d6a0b8fa38-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:41.016815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:41.016718 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c"] Oct 30 23:48:41.020849 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:41.020825 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6cb71-6c77759bdc-m7r5c"] Oct 30 23:48:41.818860 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:41.818826 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj"] Oct 30 23:48:41.819662 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:41.819631 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" containerID="cri-o://5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f" gracePeriod=30 Oct 30 23:48:41.927042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:41.926181 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs"] Oct 30 23:48:41.927042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:41.926427 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" podUID="3d62150f-cb32-4975-a88e-70b671493882" containerName="istio-proxy" containerID="cri-o://43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab" gracePeriod=30 Oct 30 23:48:42.270538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:42.270477 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq"] Oct 30 23:48:42.270749 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:42.270726 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" podUID="6474f60e-9d99-45e5-830d-4a668dbd41bb" containerName="istio-proxy" containerID="cri-o://d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c" gracePeriod=30 Oct 30 23:48:42.607269 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:42.607233 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd6943f-aabc-4990-be83-d7d6a0b8fa38" path="/var/lib/kubelet/pods/6cd6943f-aabc-4990-be83-d7d6a0b8fa38/volumes" Oct 30 23:48:47.432530 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.431272 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:48:47.437814 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.437788 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:48:47.522537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.522034 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-token\" (UniqueName: \"kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-token\") pod \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " Oct 30 23:48:47.522537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.522084 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istiod-ca-cert\") pod \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " Oct 30 23:48:47.522537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.522127 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-envoy\") pod \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " Oct 30 23:48:47.522537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.522239 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-credential-socket\") pod \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " Oct 30 23:48:47.522537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.522269 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-certs\") pod \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " Oct 30 23:48:47.522537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.522317 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-kube-api-access-9ms64\" (UniqueName: \"kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-kube-api-access-9ms64\") pod \"3d62150f-cb32-4975-a88e-70b671493882\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " Oct 30 23:48:47.522537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.522382 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-credential-socket\") pod \"3d62150f-cb32-4975-a88e-70b671493882\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " Oct 30 23:48:47.522537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.522437 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d62150f-cb32-4975-a88e-70b671493882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d62150f-cb32-4975-a88e-70b671493882-istio-podinfo\") pod \"3d62150f-cb32-4975-a88e-70b671493882\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " Oct 30 23:48:47.523761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.522895 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-credential-socket" pod "0cdbcc63-7336-46ed-99bc-0e3386f53da5" (UID: "0cdbcc63-7336-46ed-99bc-0e3386f53da5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.523761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.523298 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istiod-ca-cert" pod "0cdbcc63-7336-46ed-99bc-0e3386f53da5" (UID: "0cdbcc63-7336-46ed-99bc-0e3386f53da5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:48:47.524941 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.524908 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-istio-token\" (UniqueName: \"kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-istio-token\") pod \"3d62150f-cb32-4975-a88e-70b671493882\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " Oct 30 23:48:47.525070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.525054 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-socket\") pod \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " Oct 30 23:48:47.525128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.525107 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-kube-api-access-4rtwn\" (UniqueName: \"kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-kube-api-access-4rtwn\") pod \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " Oct 30 23:48:47.525183 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.525139 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d62150f-cb32-4975-a88e-70b671493882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d62150f-cb32-4975-a88e-70b671493882-istiod-ca-cert\") pod \"3d62150f-cb32-4975-a88e-70b671493882\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " Oct 30 23:48:47.525237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.525192 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-socket\") pod \"3d62150f-cb32-4975-a88e-70b671493882\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " Oct 30 23:48:47.525299 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.525284 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-data\") pod \"3d62150f-cb32-4975-a88e-70b671493882\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " Oct 30 23:48:47.525363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.525349 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-certs\") pod \"3d62150f-cb32-4975-a88e-70b671493882\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " Oct 30 23:48:47.525431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.525417 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-data\") pod \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " Oct 30 23:48:47.526623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.526598 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-envoy\") pod \"3d62150f-cb32-4975-a88e-70b671493882\" (UID: \"3d62150f-cb32-4975-a88e-70b671493882\") " Oct 30 23:48:47.526755 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.526639 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-podinfo\") pod \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\" (UID: \"0cdbcc63-7336-46ed-99bc-0e3386f53da5\") " Oct 30 23:48:47.528233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.527723 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-data" pod "3d62150f-cb32-4975-a88e-70b671493882" (UID: "3d62150f-cb32-4975-a88e-70b671493882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.528233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.527952 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.528233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.527972 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.528233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.527987 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.528826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.528800 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-socket" pod "0cdbcc63-7336-46ed-99bc-0e3386f53da5" (UID: "0cdbcc63-7336-46ed-99bc-0e3386f53da5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.534767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.534739 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-certs" pod "0cdbcc63-7336-46ed-99bc-0e3386f53da5" (UID: "0cdbcc63-7336-46ed-99bc-0e3386f53da5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.534932 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.534868 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-credential-socket" pod "3d62150f-cb32-4975-a88e-70b671493882" (UID: "3d62150f-cb32-4975-a88e-70b671493882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.537388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.537366 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-data" pod "0cdbcc63-7336-46ed-99bc-0e3386f53da5" (UID: "0cdbcc63-7336-46ed-99bc-0e3386f53da5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.538316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.538293 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-certs" pod "3d62150f-cb32-4975-a88e-70b671493882" (UID: "3d62150f-cb32-4975-a88e-70b671493882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.542136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.539681 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d62150f-cb32-4975-a88e-70b671493882-istiod-ca-cert" pod "3d62150f-cb32-4975-a88e-70b671493882" (UID: "3d62150f-cb32-4975-a88e-70b671493882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:48:47.542136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.539907 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-socket" pod "3d62150f-cb32-4975-a88e-70b671493882" (UID: "3d62150f-cb32-4975-a88e-70b671493882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.545513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.544133 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d62150f-cb32-4975-a88e-70b671493882-istio-podinfo" pod "3d62150f-cb32-4975-a88e-70b671493882" (UID: "3d62150f-cb32-4975-a88e-70b671493882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:48:47.545513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.545112 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-istio-token" pod "3d62150f-cb32-4975-a88e-70b671493882" (UID: "3d62150f-cb32-4975-a88e-70b671493882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:48:47.545513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.545212 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-kube-api-access-4rtwn" pod "0cdbcc63-7336-46ed-99bc-0e3386f53da5" (UID: "0cdbcc63-7336-46ed-99bc-0e3386f53da5"). InnerVolumeSpecName "kube-api-access-4rtwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:48:47.545729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.545589 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-envoy" pod "0cdbcc63-7336-46ed-99bc-0e3386f53da5" (UID: "0cdbcc63-7336-46ed-99bc-0e3386f53da5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.546668 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.546634 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-kube-api-access-9ms64" pod "3d62150f-cb32-4975-a88e-70b671493882" (UID: "3d62150f-cb32-4975-a88e-70b671493882"). InnerVolumeSpecName "kube-api-access-9ms64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:48:47.546771 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.546744 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-token" pod "0cdbcc63-7336-46ed-99bc-0e3386f53da5" (UID: "0cdbcc63-7336-46ed-99bc-0e3386f53da5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:48:47.547709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.547684 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-podinfo" pod "0cdbcc63-7336-46ed-99bc-0e3386f53da5" (UID: "0cdbcc63-7336-46ed-99bc-0e3386f53da5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:48:47.549337 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.549314 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-envoy" pod "3d62150f-cb32-4975-a88e-70b671493882" (UID: "3d62150f-cb32-4975-a88e-70b671493882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.619975 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.619946 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 30 23:48:47.620891 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.620867 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:48:47.630436 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630409 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630441 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ms64\" (UniqueName: \"kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-kube-api-access-9ms64\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630474 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630489 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d62150f-cb32-4975-a88e-70b671493882-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630502 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d62150f-cb32-4975-a88e-70b671493882-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630514 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630528 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rtwn\" (UniqueName: \"kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-kube-api-access-4rtwn\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630541 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d62150f-cb32-4975-a88e-70b671493882-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630554 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630566 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630578 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630601 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d62150f-cb32-4975-a88e-70b671493882-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630614 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630626 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.630942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.630639 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdbcc63-7336-46ed-99bc-0e3386f53da5-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.731797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.731706 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-podinfo\") pod \"6474f60e-9d99-45e5-830d-4a668dbd41bb\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " Oct 30 23:48:47.731966 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.731858 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-data\") pod \"6474f60e-9d99-45e5-830d-4a668dbd41bb\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " Oct 30 23:48:47.731966 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.731897 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-credential-socket\") pod \"6474f60e-9d99-45e5-830d-4a668dbd41bb\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " Oct 30 23:48:47.732085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.732036 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-socket\") pod \"6474f60e-9d99-45e5-830d-4a668dbd41bb\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.732151 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-envoy\") pod \"6474f60e-9d99-45e5-830d-4a668dbd41bb\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.732199 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-kube-api-access-d4sgl\" (UniqueName: \"kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-kube-api-access-d4sgl\") pod \"6474f60e-9d99-45e5-830d-4a668dbd41bb\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.732266 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-certs\") pod \"6474f60e-9d99-45e5-830d-4a668dbd41bb\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.732275 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-data" pod "6474f60e-9d99-45e5-830d-4a668dbd41bb" (UID: "6474f60e-9d99-45e5-830d-4a668dbd41bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.732310 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-credential-socket" pod "6474f60e-9d99-45e5-830d-4a668dbd41bb" (UID: "6474f60e-9d99-45e5-830d-4a668dbd41bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.732423 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-token\" (UniqueName: \"kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-token\") pod \"6474f60e-9d99-45e5-830d-4a668dbd41bb\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.732479 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6474f60e-9d99-45e5-830d-4a668dbd41bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6474f60e-9d99-45e5-830d-4a668dbd41bb-istiod-ca-cert\") pod \"6474f60e-9d99-45e5-830d-4a668dbd41bb\" (UID: \"6474f60e-9d99-45e5-830d-4a668dbd41bb\") " Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.732629 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-certs" pod "6474f60e-9d99-45e5-830d-4a668dbd41bb" (UID: "6474f60e-9d99-45e5-830d-4a668dbd41bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.732680 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-socket" pod "6474f60e-9d99-45e5-830d-4a668dbd41bb" (UID: "6474f60e-9d99-45e5-830d-4a668dbd41bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.733129 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6474f60e-9d99-45e5-830d-4a668dbd41bb-istiod-ca-cert" pod "6474f60e-9d99-45e5-830d-4a668dbd41bb" (UID: "6474f60e-9d99-45e5-830d-4a668dbd41bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.733989 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.734011 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.734026 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.734040 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.734992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.734052 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6474f60e-9d99-45e5-830d-4a668dbd41bb-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.735622 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.735028 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-envoy" pod "6474f60e-9d99-45e5-830d-4a668dbd41bb" (UID: "6474f60e-9d99-45e5-830d-4a668dbd41bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:47.735622 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.735072 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-podinfo" pod "6474f60e-9d99-45e5-830d-4a668dbd41bb" (UID: "6474f60e-9d99-45e5-830d-4a668dbd41bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:48:47.736815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.736792 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-kube-api-access-d4sgl" pod "6474f60e-9d99-45e5-830d-4a668dbd41bb" (UID: "6474f60e-9d99-45e5-830d-4a668dbd41bb"). InnerVolumeSpecName "kube-api-access-d4sgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:48:47.737266 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.737246 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-token" pod "6474f60e-9d99-45e5-830d-4a668dbd41bb" (UID: "6474f60e-9d99-45e5-830d-4a668dbd41bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:48:47.835682 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.835649 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.835682 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.835681 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4sgl\" (UniqueName: \"kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-kube-api-access-d4sgl\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.835879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.835694 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:47.835879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:47.835706 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6474f60e-9d99-45e5-830d-4a668dbd41bb-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:48.002709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.002624 7570 generic.go:356] "Generic (PLEG): container finished" podID="3d62150f-cb32-4975-a88e-70b671493882" containerID="43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab" exitCode=0 Oct 30 23:48:48.002709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.002676 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" Oct 30 23:48:48.002992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.002726 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" event={"ID":"3d62150f-cb32-4975-a88e-70b671493882","Type":"ContainerDied","Data":"43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab"} Oct 30 23:48:48.002992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.002795 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs" event={"ID":"3d62150f-cb32-4975-a88e-70b671493882","Type":"ContainerDied","Data":"b32ce357d11afa23c6e344b5646bbf6cd2a06ed720268c0ca0151ed391b9e92a"} Oct 30 23:48:48.002992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.002822 7570 scope.go:122] "RemoveContainer" containerID="43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab" Oct 30 23:48:48.004502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.004436 7570 generic.go:356] "Generic (PLEG): container finished" podID="6474f60e-9d99-45e5-830d-4a668dbd41bb" containerID="d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c" exitCode=0 Oct 30 23:48:48.004612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.004538 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" Oct 30 23:48:48.004612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.004575 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" event={"ID":"6474f60e-9d99-45e5-830d-4a668dbd41bb","Type":"ContainerDied","Data":"d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c"} Oct 30 23:48:48.004739 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.004613 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq" event={"ID":"6474f60e-9d99-45e5-830d-4a668dbd41bb","Type":"ContainerDied","Data":"c374cc462a0c5664b46b0b45969152f114104a7f39e5752bc3cf052e34786371"} Oct 30 23:48:48.006311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.006289 7570 generic.go:356] "Generic (PLEG): container finished" podID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerID="5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f" exitCode=0 Oct 30 23:48:48.006522 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.006504 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" Oct 30 23:48:48.006616 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.006524 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" event={"ID":"0cdbcc63-7336-46ed-99bc-0e3386f53da5","Type":"ContainerDied","Data":"5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f"} Oct 30 23:48:48.006616 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.006563 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj" event={"ID":"0cdbcc63-7336-46ed-99bc-0e3386f53da5","Type":"ContainerDied","Data":"804a09e7e978f598f9971e63491ec11b5ab90aaee30810692b5479c00794c197"} Oct 30 23:48:48.021883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.021371 7570 scope.go:122] "RemoveContainer" containerID="43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab" Oct 30 23:48:48.022358 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:48:48.022303 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab\": container with ID starting with 43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab not found: ID does not exist" containerID="43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab" Oct 30 23:48:48.022528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.022373 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab"} err="failed to get container status \"43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab\": rpc error: code = NotFound desc = could not find container \"43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab\": container with ID starting with 43830bd71f00753c93a27b740135a28d3438bf250076c4261b345dfde096aaab not found: ID does not exist" Oct 30 23:48:48.022528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.022396 7570 scope.go:122] "RemoveContainer" containerID="d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c" Oct 30 23:48:48.040198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.040166 7570 scope.go:122] "RemoveContainer" containerID="d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c" Oct 30 23:48:48.042392 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:48:48.042359 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c\": container with ID starting with d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c not found: ID does not exist" containerID="d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c" Oct 30 23:48:48.042514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.042400 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c"} err="failed to get container status \"d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c\": rpc error: code = NotFound desc = could not find container \"d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c\": container with ID starting with d943f90eb46ba73fd6635d00d31a97995fa4994195a748ee23cd25d2dc1b249c not found: ID does not exist" Oct 30 23:48:48.042514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.042424 7570 scope.go:122] "RemoveContainer" containerID="5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f" Oct 30 23:48:48.053039 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.053015 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj"] Oct 30 23:48:48.058584 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.058557 7570 scope.go:122] "RemoveContainer" containerID="5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f" Oct 30 23:48:48.059080 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:48:48.059059 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f\": container with ID starting with 5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f not found: ID does not exist" containerID="5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f" Oct 30 23:48:48.059196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.059091 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f"} err="failed to get container status \"5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f\": rpc error: code = NotFound desc = could not find container \"5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f\": container with ID starting with 5ce0dbd9c82529ee0c0ef599fd0a44045295562c04425e07e049891acf07cf8f not found: ID does not exist" Oct 30 23:48:48.064178 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.064151 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-fhjkj"] Oct 30 23:48:48.085377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.085353 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq"] Oct 30 23:48:48.089087 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.089063 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-j46qq"] Oct 30 23:48:48.100344 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.100292 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs"] Oct 30 23:48:48.104062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.104038 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-cgvbs"] Oct 30 23:48:48.192530 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.192494 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8"] Oct 30 23:48:48.192786 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.192760 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" containerID="cri-o://0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6" gracePeriod=30 Oct 30 23:48:48.630432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.630385 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" path="/var/lib/kubelet/pods/0cdbcc63-7336-46ed-99bc-0e3386f53da5/volumes" Oct 30 23:48:48.631279 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.631257 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d62150f-cb32-4975-a88e-70b671493882" path="/var/lib/kubelet/pods/3d62150f-cb32-4975-a88e-70b671493882/volumes" Oct 30 23:48:48.631954 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:48.631933 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6474f60e-9d99-45e5-830d-4a668dbd41bb" path="/var/lib/kubelet/pods/6474f60e-9d99-45e5-830d-4a668dbd41bb/volumes" Oct 30 23:48:50.677068 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:50.677037 7570 scope.go:122] "RemoveContainer" containerID="9ae5523100a396fb2399cff176642e92f6fbf5f7c353059c3bacae37fa284b2d" Oct 30 23:48:50.694368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:50.694346 7570 scope.go:122] "RemoveContainer" containerID="9c13400c2777f77ae5d445ac2913dff5775d73a94f55442fa3e1968275bae38c" Oct 30 23:48:50.705728 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:50.705705 7570 scope.go:122] "RemoveContainer" containerID="876521141f9b121dff6bd4728a57ef29a07060e89105a3ef7e51948fbaa6d73e" Oct 30 23:48:50.717000 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:50.716981 7570 scope.go:122] "RemoveContainer" containerID="21d7b02cbb0da5227ca3463e2496a43a766a74b2b444c25926e011b95f38766d" Oct 30 23:48:50.727932 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:50.727904 7570 scope.go:122] "RemoveContainer" containerID="39ae65978ca64880fb0425e946c9df7d7000b3f3602ebd44712fca370c145ea4" Oct 30 23:48:50.738494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:50.738477 7570 scope.go:122] "RemoveContainer" containerID="f39c679031f9aafb52ab04d1363de081f4a29a2abb51f84b5d35d53eb1335b15" Oct 30 23:48:50.748827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:50.748804 7570 scope.go:122] "RemoveContainer" containerID="f5ba846fca363916029b861af4486c90fdc98342c7fd3c4872d8d350d0650c0c" Oct 30 23:48:50.759272 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:50.759155 7570 scope.go:122] "RemoveContainer" containerID="b56b05b193a948af227f890ca948b560cc2d06f9b102cf29c19182485a45487f" Oct 30 23:48:50.771935 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:50.771915 7570 scope.go:122] "RemoveContainer" containerID="d6098a99f1a1102829afc800764271caa3b2afad2623cd9216ad27f435b35b51" Oct 30 23:48:51.946904 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:51.946878 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 30 23:48:53.505456 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.505427 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:48:53.615991 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.615954 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-credential-socket\") pod \"71246733-216c-48db-8261-cef843d9b6b4\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " Oct 30 23:48:53.616165 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.616109 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71246733-216c-48db-8261-cef843d9b6b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71246733-216c-48db-8261-cef843d9b6b4-istio-podinfo\") pod \"71246733-216c-48db-8261-cef843d9b6b4\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " Oct 30 23:48:53.616236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.616170 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-socket\") pod \"71246733-216c-48db-8261-cef843d9b6b4\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " Oct 30 23:48:53.616322 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.616232 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-credential-socket" pod "71246733-216c-48db-8261-cef843d9b6b4" (UID: "71246733-216c-48db-8261-cef843d9b6b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:53.616322 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.616306 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-istio-token\" (UniqueName: \"kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-istio-token\") pod \"71246733-216c-48db-8261-cef843d9b6b4\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " Oct 30 23:48:53.616440 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.616362 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-data\") pod \"71246733-216c-48db-8261-cef843d9b6b4\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " Oct 30 23:48:53.616520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.616442 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-envoy\") pod \"71246733-216c-48db-8261-cef843d9b6b4\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " Oct 30 23:48:53.616520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.616452 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-socket" pod "71246733-216c-48db-8261-cef843d9b6b4" (UID: "71246733-216c-48db-8261-cef843d9b6b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:53.616681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.616554 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-certs\") pod \"71246733-216c-48db-8261-cef843d9b6b4\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " Oct 30 23:48:53.616772 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.616689 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-kube-api-access-2xcdx\" (UniqueName: \"kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-kube-api-access-2xcdx\") pod \"71246733-216c-48db-8261-cef843d9b6b4\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " Oct 30 23:48:53.616772 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.616729 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71246733-216c-48db-8261-cef843d9b6b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71246733-216c-48db-8261-cef843d9b6b4-istiod-ca-cert\") pod \"71246733-216c-48db-8261-cef843d9b6b4\" (UID: \"71246733-216c-48db-8261-cef843d9b6b4\") " Oct 30 23:48:53.616877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.616857 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-data" pod "71246733-216c-48db-8261-cef843d9b6b4" (UID: "71246733-216c-48db-8261-cef843d9b6b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:53.617093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.616985 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-certs" pod "71246733-216c-48db-8261-cef843d9b6b4" (UID: "71246733-216c-48db-8261-cef843d9b6b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:53.617317 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.617253 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71246733-216c-48db-8261-cef843d9b6b4-istiod-ca-cert" pod "71246733-216c-48db-8261-cef843d9b6b4" (UID: "71246733-216c-48db-8261-cef843d9b6b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:48:53.618066 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.618039 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:53.618181 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.618069 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71246733-216c-48db-8261-cef843d9b6b4-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:53.618181 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.618085 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:53.618181 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.618100 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:53.618181 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.618115 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:53.618816 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.618792 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71246733-216c-48db-8261-cef843d9b6b4-istio-podinfo" pod "71246733-216c-48db-8261-cef843d9b6b4" (UID: "71246733-216c-48db-8261-cef843d9b6b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:48:53.618924 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.618835 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-istio-token" pod "71246733-216c-48db-8261-cef843d9b6b4" (UID: "71246733-216c-48db-8261-cef843d9b6b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:48:53.619351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.619332 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-envoy" pod "71246733-216c-48db-8261-cef843d9b6b4" (UID: "71246733-216c-48db-8261-cef843d9b6b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:48:53.620212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.620188 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-kube-api-access-2xcdx" pod "71246733-216c-48db-8261-cef843d9b6b4" (UID: "71246733-216c-48db-8261-cef843d9b6b4"). InnerVolumeSpecName "kube-api-access-2xcdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:48:53.720395 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.720313 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71246733-216c-48db-8261-cef843d9b6b4-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:53.720395 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.720338 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:53.720395 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.720349 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71246733-216c-48db-8261-cef843d9b6b4-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:53.720395 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.720358 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xcdx\" (UniqueName: \"kubernetes.io/projected/71246733-216c-48db-8261-cef843d9b6b4-kube-api-access-2xcdx\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:48:53.767529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.767496 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh"] Oct 30 23:48:53.768366 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768345 7570 cpu_manager.go:414] "containerMap: removing container" podUID="3d62150f-cb32-4975-a88e-70b671493882" containerName="istio-proxy" Oct 30 23:48:53.768366 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768365 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d62150f-cb32-4975-a88e-70b671493882" containerName="istio-proxy" Oct 30 23:48:53.768562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768380 7570 cpu_manager.go:414] "containerMap: removing container" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" Oct 30 23:48:53.768562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768388 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" Oct 30 23:48:53.768562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768402 7570 cpu_manager.go:414] "containerMap: removing container" podUID="6cd6943f-aabc-4990-be83-d7d6a0b8fa38" containerName="maas-api" Oct 30 23:48:53.768562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768409 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd6943f-aabc-4990-be83-d7d6a0b8fa38" containerName="maas-api" Oct 30 23:48:53.768562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768452 7570 cpu_manager.go:414] "containerMap: removing container" podUID="6474f60e-9d99-45e5-830d-4a668dbd41bb" containerName="istio-proxy" Oct 30 23:48:53.768562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768475 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6474f60e-9d99-45e5-830d-4a668dbd41bb" containerName="istio-proxy" Oct 30 23:48:53.768562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768488 7570 cpu_manager.go:414] "containerMap: removing container" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" Oct 30 23:48:53.768562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768494 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" Oct 30 23:48:53.768562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768514 7570 cpu_manager.go:414] "containerMap: removing container" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" Oct 30 23:48:53.768562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768521 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" Oct 30 23:48:53.769047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768719 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6474f60e-9d99-45e5-830d-4a668dbd41bb" containerName="istio-proxy" Oct 30 23:48:53.769047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768738 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" Oct 30 23:48:53.769047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768749 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="71246733-216c-48db-8261-cef843d9b6b4" containerName="istio-proxy" Oct 30 23:48:53.769047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768768 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d62150f-cb32-4975-a88e-70b671493882" containerName="istio-proxy" Oct 30 23:48:53.769047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768777 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cdbcc63-7336-46ed-99bc-0e3386f53da5" containerName="istio-proxy" Oct 30 23:48:53.769047 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.768787 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd6943f-aabc-4990-be83-d7d6a0b8fa38" containerName="maas-api" Oct 30 23:48:53.773256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.773223 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:53.780996 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.780961 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-885999-serving-cert\"" Oct 30 23:48:53.793990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.793956 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh"] Oct 30 23:48:53.952860 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.939193 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dljfh\" (UniqueName: \"kubernetes.io/projected/eedd4bdb-1495-47e6-884d-5ce26a05e72b-kube-api-access-dljfh\") pod \"maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:53.952860 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.939287 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:53.952860 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.939421 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eedd4bdb-1495-47e6-884d-5ce26a05e72b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:53.952860 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:53.939616 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:54.042309 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.042211 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dljfh\" (UniqueName: \"kubernetes.io/projected/eedd4bdb-1495-47e6-884d-5ce26a05e72b-kube-api-access-dljfh\") pod \"maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:54.042725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.042693 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:54.042872 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.042850 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eedd4bdb-1495-47e6-884d-5ce26a05e72b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:54.043017 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.043000 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:54.046265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.046230 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:54.046371 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.046279 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:54.046826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.046801 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eedd4bdb-1495-47e6-884d-5ce26a05e72b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:54.054935 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.054913 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dljfh\" (UniqueName: \"kubernetes.io/projected/eedd4bdb-1495-47e6-884d-5ce26a05e72b-kube-api-access-dljfh\") pod \"maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:54.097042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.097009 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:54.120654 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.120626 7570 generic.go:356] "Generic (PLEG): container finished" podID="71246733-216c-48db-8261-cef843d9b6b4" containerID="0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6" exitCode=0 Oct 30 23:48:54.120767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.120682 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" Oct 30 23:48:54.120767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.120685 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" event={"ID":"71246733-216c-48db-8261-cef843d9b6b4","Type":"ContainerDied","Data":"0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6"} Oct 30 23:48:54.120767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.120747 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8" event={"ID":"71246733-216c-48db-8261-cef843d9b6b4","Type":"ContainerDied","Data":"c17f60e4dbe08e11e7e6d1610838c085e3c27b01a9bbed903e8857389eecf85c"} Oct 30 23:48:54.120912 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.120772 7570 scope.go:122] "RemoveContainer" containerID="0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6" Oct 30 23:48:54.132883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.132861 7570 scope.go:122] "RemoveContainer" containerID="18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77" Oct 30 23:48:54.146405 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.146340 7570 scope.go:122] "RemoveContainer" containerID="0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6" Oct 30 23:48:54.147073 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:48:54.146930 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6\": container with ID starting with 0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6 not found: ID does not exist" containerID="0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6" Oct 30 23:48:54.147073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.146966 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6"} err="failed to get container status \"0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6\": rpc error: code = NotFound desc = could not find container \"0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6\": container with ID starting with 0fd7fa44e0f467752e28999f3d3a4833c814f6b927c188b6a122ae4c8ce542c6 not found: ID does not exist" Oct 30 23:48:54.147073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.146992 7570 scope.go:122] "RemoveContainer" containerID="18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77" Oct 30 23:48:54.147376 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:48:54.147351 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77\": container with ID starting with 18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77 not found: ID does not exist" containerID="18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77" Oct 30 23:48:54.147436 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.147387 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77"} err="failed to get container status \"18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77\": rpc error: code = NotFound desc = could not find container \"18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77\": container with ID starting with 18c7c52aeb7b778c00ea57e592a2aaf31df97b68fdf8e5512bcda4e70a8cff77 not found: ID does not exist" Oct 30 23:48:54.153060 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.153038 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8"] Oct 30 23:48:54.163074 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.163043 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b6cb71-openshift-default-dfb7bf949-dc5b8"] Oct 30 23:48:54.467729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.466522 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh"] Oct 30 23:48:54.605061 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:54.605019 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71246733-216c-48db-8261-cef843d9b6b4" path="/var/lib/kubelet/pods/71246733-216c-48db-8261-cef843d9b6b4/volumes" Oct 30 23:48:55.168486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:55.165749 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" event={"ID":"eedd4bdb-1495-47e6-884d-5ce26a05e72b","Type":"ContainerStarted","Data":"c15f4cb49b41c79f4a364ec6d09bb2a1293706fbb896c110d39fd96de690d417"} Oct 30 23:48:56.050531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.050496 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5"] Oct 30 23:48:56.056841 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.056815 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.074936 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.074885 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1bc837-serving-cert\"" Oct 30 23:48:56.076264 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.076239 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5"] Oct 30 23:48:56.137683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.137649 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-maas-api-tls\") pod \"maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.137893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.137799 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e6c870-08ba-4b3c-a26d-45397d5ca730-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.137962 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.137893 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2qwd\" (UniqueName: \"kubernetes.io/projected/06e6c870-08ba-4b3c-a26d-45397d5ca730-kube-api-access-z2qwd\") pod \"maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.138059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.138045 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-metrics-certs\") pod \"maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.175815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.175763 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" event={"ID":"eedd4bdb-1495-47e6-884d-5ce26a05e72b","Type":"ContainerStarted","Data":"e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358"} Oct 30 23:48:56.176012 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.175841 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:48:56.221615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.221552 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podStartSLOduration=2.660036415 podStartE2EDuration="3.221530654s" totalImagesPullingTime="561.494239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:56.216737769 +0000 UTC m=+1747.735738859" watchObservedRunningTime="2026-10-30 23:48:56.221530654 +0000 UTC m=+1747.740531745" Oct 30 23:48:56.239863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.239827 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e6c870-08ba-4b3c-a26d-45397d5ca730-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.240064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.239920 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2qwd\" (UniqueName: \"kubernetes.io/projected/06e6c870-08ba-4b3c-a26d-45397d5ca730-kube-api-access-z2qwd\") pod \"maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.240665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.240104 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-metrics-certs\") pod \"maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.240665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.240305 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-maas-api-tls\") pod \"maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.240665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.240497 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e6c870-08ba-4b3c-a26d-45397d5ca730-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.243702 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.243679 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-metrics-certs\") pod \"maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.243702 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.243688 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-maas-api-tls\") pod \"maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.253035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.253002 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2qwd\" (UniqueName: \"kubernetes.io/projected/06e6c870-08ba-4b3c-a26d-45397d5ca730-kube-api-access-z2qwd\") pod \"maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.383896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.383859 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:56.878205 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:56.878162 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5"] Oct 30 23:48:56.879395 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:48:56.879369 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e6c870_08ba_4b3c_a26d_45397d5ca730.slice/crio-a917798093e9458cf04e54e4797867c08c1f8671347231299a14dceb5da18525 WatchSource:0}: Error finding container a917798093e9458cf04e54e4797867c08c1f8671347231299a14dceb5da18525: Status 404 returned error can't find the container with id a917798093e9458cf04e54e4797867c08c1f8671347231299a14dceb5da18525 Oct 30 23:48:57.198489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:57.196721 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" event={"ID":"06e6c870-08ba-4b3c-a26d-45397d5ca730","Type":"ContainerStarted","Data":"a917798093e9458cf04e54e4797867c08c1f8671347231299a14dceb5da18525"} Oct 30 23:48:58.238780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:58.238740 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" event={"ID":"06e6c870-08ba-4b3c-a26d-45397d5ca730","Type":"ContainerStarted","Data":"5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51"} Oct 30 23:48:58.252203 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:48:58.252171 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:48:59.419686 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:48:59.419636 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:49:02.202721 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.202692 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:49:02.212480 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.212421 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podStartSLOduration=5.690336578 podStartE2EDuration="6.212406266s" totalImagesPullingTime="522.069688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:58.27877404 +0000 UTC m=+1749.797775130" watchObservedRunningTime="2026-10-30 23:49:02.212406266 +0000 UTC m=+1753.731407356" Oct 30 23:49:02.213430 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.213412 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s"] Oct 30 23:49:02.218209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.218195 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.222710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.222685 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6a162557-serving-cert\"" Oct 30 23:49:02.229153 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.229124 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s"] Oct 30 23:49:02.331515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.331484 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmwcv\" (UniqueName: \"kubernetes.io/projected/4688133b-ced0-47cc-aac2-e0cf0237ee52-kube-api-access-hmwcv\") pod \"maas-api-e2e-mt-6a162557-55c4486749-qp69s\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.331713 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.331629 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688133b-ced0-47cc-aac2-e0cf0237ee52-openshift-service-ca\") pod \"maas-api-e2e-mt-6a162557-55c4486749-qp69s\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.331784 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.331734 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-metrics-certs\") pod \"maas-api-e2e-mt-6a162557-55c4486749-qp69s\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.331899 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.331885 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-maas-api-tls\") pod \"maas-api-e2e-mt-6a162557-55c4486749-qp69s\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.468164 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.467848 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-maas-api-tls\") pod \"maas-api-e2e-mt-6a162557-55c4486749-qp69s\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.468164 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.468149 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmwcv\" (UniqueName: \"kubernetes.io/projected/4688133b-ced0-47cc-aac2-e0cf0237ee52-kube-api-access-hmwcv\") pod \"maas-api-e2e-mt-6a162557-55c4486749-qp69s\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.468416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.468264 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688133b-ced0-47cc-aac2-e0cf0237ee52-openshift-service-ca\") pod \"maas-api-e2e-mt-6a162557-55c4486749-qp69s\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.468416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.468359 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-metrics-certs\") pod \"maas-api-e2e-mt-6a162557-55c4486749-qp69s\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.480532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.480155 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688133b-ced0-47cc-aac2-e0cf0237ee52-openshift-service-ca\") pod \"maas-api-e2e-mt-6a162557-55c4486749-qp69s\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.485690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.485502 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-maas-api-tls\") pod \"maas-api-e2e-mt-6a162557-55c4486749-qp69s\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.495987 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.495957 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-metrics-certs\") pod \"maas-api-e2e-mt-6a162557-55c4486749-qp69s\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.498431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.498403 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmwcv\" (UniqueName: \"kubernetes.io/projected/4688133b-ced0-47cc-aac2-e0cf0237ee52-kube-api-access-hmwcv\") pod \"maas-api-e2e-mt-6a162557-55c4486749-qp69s\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.544155 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.544116 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:02.850443 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:49:02.850406 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4688133b_ced0_47cc_aac2_e0cf0237ee52.slice/crio-8ba4ff04e71c48c24fa6655fb5f404187cad3ec935c377a8a6113dab43851839 WatchSource:0}: Error finding container 8ba4ff04e71c48c24fa6655fb5f404187cad3ec935c377a8a6113dab43851839: Status 404 returned error can't find the container with id 8ba4ff04e71c48c24fa6655fb5f404187cad3ec935c377a8a6113dab43851839 Oct 30 23:49:02.850719 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:02.850695 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s"] Oct 30 23:49:03.332544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:03.318049 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" event={"ID":"4688133b-ced0-47cc-aac2-e0cf0237ee52","Type":"ContainerStarted","Data":"8ba4ff04e71c48c24fa6655fb5f404187cad3ec935c377a8a6113dab43851839"} Oct 30 23:49:04.329040 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:04.329001 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" event={"ID":"4688133b-ced0-47cc-aac2-e0cf0237ee52","Type":"ContainerStarted","Data":"8580508d5ca11d86b887d1b6d100d6b41c8ee355cdc848472a3202440831f137"} Oct 30 23:49:04.329217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:04.329063 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:04.350678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:04.350622 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" podStartSLOduration=1.863087808 podStartE2EDuration="2.350608934s" totalImagesPullingTime="487.521126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:04.348868753 +0000 UTC m=+1755.867869845" watchObservedRunningTime="2026-10-30 23:49:04.350608934 +0000 UTC m=+1755.869610038" Oct 30 23:49:05.261123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:05.261092 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:49:10.344495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:10.344433 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:12.024865 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:12.024832 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl"] Oct 30 23:49:12.025367 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:12.025062 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" podUID="792ae9e3-7af7-4767-900f-409a34cc02ac" containerName="istio-proxy" containerID="cri-o://7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f" gracePeriod=30 Oct 30 23:49:12.030103 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:12.030075 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24"] Oct 30 23:49:12.030634 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:12.030579 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerName="istio-proxy" containerID="cri-o://162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1" gracePeriod=30 Oct 30 23:49:12.036325 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:12.036300 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5"] Oct 30 23:49:12.036576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:12.036551 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" containerID="cri-o://dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272" gracePeriod=30 Oct 30 23:49:12.314808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:12.312815 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6"] Oct 30 23:49:12.315302 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:12.315269 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" podUID="0c029726-9be9-4ac8-a268-8761a4499b87" containerName="istio-proxy" containerID="cri-o://73e9a637495396add78640f3ed8d7c33b22555c9bb55f7223f94a09d6628bc94" gracePeriod=30 Oct 30 23:49:17.358520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.358491 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:49:17.436555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.436525 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-kube-api-access-sf4s2\" (UniqueName: \"kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-kube-api-access-sf4s2\") pod \"792ae9e3-7af7-4767-900f-409a34cc02ac\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " Oct 30 23:49:17.436748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.436640 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-data\") pod \"792ae9e3-7af7-4767-900f-409a34cc02ac\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " Oct 30 23:49:17.436748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.436690 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/792ae9e3-7af7-4767-900f-409a34cc02ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792ae9e3-7af7-4767-900f-409a34cc02ac-istiod-ca-cert\") pod \"792ae9e3-7af7-4767-900f-409a34cc02ac\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " Oct 30 23:49:17.436748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.436742 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-socket\") pod \"792ae9e3-7af7-4767-900f-409a34cc02ac\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " Oct 30 23:49:17.436925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.436760 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-podinfo\") pod \"792ae9e3-7af7-4767-900f-409a34cc02ac\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " Oct 30 23:49:17.436925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.436907 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-token\" (UniqueName: \"kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-token\") pod \"792ae9e3-7af7-4767-900f-409a34cc02ac\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " Oct 30 23:49:17.437044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.436952 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-credential-socket\") pod \"792ae9e3-7af7-4767-900f-409a34cc02ac\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " Oct 30 23:49:17.437044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.437003 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-data" pod "792ae9e3-7af7-4767-900f-409a34cc02ac" (UID: "792ae9e3-7af7-4767-900f-409a34cc02ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.437151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.437050 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-envoy\") pod \"792ae9e3-7af7-4767-900f-409a34cc02ac\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " Oct 30 23:49:17.437151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.437091 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-certs\") pod \"792ae9e3-7af7-4767-900f-409a34cc02ac\" (UID: \"792ae9e3-7af7-4767-900f-409a34cc02ac\") " Oct 30 23:49:17.437294 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.437171 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792ae9e3-7af7-4767-900f-409a34cc02ac-istiod-ca-cert" pod "792ae9e3-7af7-4767-900f-409a34cc02ac" (UID: "792ae9e3-7af7-4767-900f-409a34cc02ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:49:17.437294 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.437261 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-credential-socket" pod "792ae9e3-7af7-4767-900f-409a34cc02ac" (UID: "792ae9e3-7af7-4767-900f-409a34cc02ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.437597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.437581 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-certs" pod "792ae9e3-7af7-4767-900f-409a34cc02ac" (UID: "792ae9e3-7af7-4767-900f-409a34cc02ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.437794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.437754 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-socket" pod "792ae9e3-7af7-4767-900f-409a34cc02ac" (UID: "792ae9e3-7af7-4767-900f-409a34cc02ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.438150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.438130 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.438230 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.438159 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.438230 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.438175 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.438230 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.438192 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792ae9e3-7af7-4767-900f-409a34cc02ac-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.438230 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.438206 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.439777 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.439750 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-kube-api-access-sf4s2" pod "792ae9e3-7af7-4767-900f-409a34cc02ac" (UID: "792ae9e3-7af7-4767-900f-409a34cc02ac"). InnerVolumeSpecName "kube-api-access-sf4s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:17.440610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.440584 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-podinfo" pod "792ae9e3-7af7-4767-900f-409a34cc02ac" (UID: "792ae9e3-7af7-4767-900f-409a34cc02ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:49:17.440876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.440849 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-envoy" pod "792ae9e3-7af7-4767-900f-409a34cc02ac" (UID: "792ae9e3-7af7-4767-900f-409a34cc02ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.444852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.444807 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-token" pod "792ae9e3-7af7-4767-900f-409a34cc02ac" (UID: "792ae9e3-7af7-4767-900f-409a34cc02ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:17.463006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.462743 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:49:17.466986 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.466962 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:49:17.508626 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.508588 7570 generic.go:356] "Generic (PLEG): container finished" podID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerID="dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272" exitCode=0 Oct 30 23:49:17.508797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.508645 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" Oct 30 23:49:17.508797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.508668 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" event={"ID":"bc946897-4a76-4a95-b7ed-c9402f9a4d42","Type":"ContainerDied","Data":"dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272"} Oct 30 23:49:17.508797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.508725 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5" event={"ID":"bc946897-4a76-4a95-b7ed-c9402f9a4d42","Type":"ContainerDied","Data":"4d41f91df84b22080743025a5491a58ef4008d9df24ec332c218cff2c5153ec8"} Oct 30 23:49:17.508797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.508756 7570 scope.go:122] "RemoveContainer" containerID="dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272" Oct 30 23:49:17.523532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.521334 7570 generic.go:356] "Generic (PLEG): container finished" podID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerID="162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1" exitCode=0 Oct 30 23:49:17.523532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.521448 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" event={"ID":"5dfe9d9e-818f-4909-a48e-555b42174d32","Type":"ContainerDied","Data":"162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1"} Oct 30 23:49:17.523532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.521511 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" event={"ID":"5dfe9d9e-818f-4909-a48e-555b42174d32","Type":"ContainerDied","Data":"19c929b25349428aa790ca140d26837c1d0d53c3206915f68c25d986ac7568e2"} Oct 30 23:49:17.524529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.524081 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24" Oct 30 23:49:17.526828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.526805 7570 generic.go:356] "Generic (PLEG): container finished" podID="792ae9e3-7af7-4767-900f-409a34cc02ac" containerID="7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f" exitCode=0 Oct 30 23:49:17.527140 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.527121 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" Oct 30 23:49:17.529508 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.527163 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" event={"ID":"792ae9e3-7af7-4767-900f-409a34cc02ac","Type":"ContainerDied","Data":"7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f"} Oct 30 23:49:17.529622 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.529541 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl" event={"ID":"792ae9e3-7af7-4767-900f-409a34cc02ac","Type":"ContainerDied","Data":"979cb4ba7f8ec82c00e285c434b8382fc1047129086878ceaa32ab683ec6fd07"} Oct 30 23:49:17.530111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.530090 7570 generic.go:356] "Generic (PLEG): container finished" podID="0c029726-9be9-4ac8-a268-8761a4499b87" containerID="73e9a637495396add78640f3ed8d7c33b22555c9bb55f7223f94a09d6628bc94" exitCode=0 Oct 30 23:49:17.530191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.530152 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" event={"ID":"0c029726-9be9-4ac8-a268-8761a4499b87","Type":"ContainerDied","Data":"73e9a637495396add78640f3ed8d7c33b22555c9bb55f7223f94a09d6628bc94"} Oct 30 23:49:17.538872 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.538848 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istiod-ca-cert\") pod \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " Oct 30 23:49:17.539086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.539059 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-data\") pod \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " Oct 30 23:49:17.539217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.539203 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dfe9d9e-818f-4909-a48e-555b42174d32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe9d9e-818f-4909-a48e-555b42174d32-istiod-ca-cert\") pod \"5dfe9d9e-818f-4909-a48e-555b42174d32\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " Oct 30 23:49:17.539327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.539305 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-kube-api-access-mdnh9\" (UniqueName: \"kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-kube-api-access-mdnh9\") pod \"5dfe9d9e-818f-4909-a48e-555b42174d32\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " Oct 30 23:49:17.539452 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.539439 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-certs\") pod \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " Oct 30 23:49:17.539655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.539640 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-kube-api-access-bjfjp\" (UniqueName: \"kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-kube-api-access-bjfjp\") pod \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " Oct 30 23:49:17.539769 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.539756 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-token\" (UniqueName: \"kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-token\") pod \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " Oct 30 23:49:17.539897 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.539885 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-podinfo\") pod \"5dfe9d9e-818f-4909-a48e-555b42174d32\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " Oct 30 23:49:17.539989 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.539976 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-certs\") pod \"5dfe9d9e-818f-4909-a48e-555b42174d32\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " Oct 30 23:49:17.540078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.540066 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-data\") pod \"5dfe9d9e-818f-4909-a48e-555b42174d32\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " Oct 30 23:49:17.540333 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.540319 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-credential-socket\") pod \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " Oct 30 23:49:17.540554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.540540 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-token\" (UniqueName: \"kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-token\") pod \"5dfe9d9e-818f-4909-a48e-555b42174d32\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " Oct 30 23:49:17.540678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.540665 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-envoy\") pod \"5dfe9d9e-818f-4909-a48e-555b42174d32\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " Oct 30 23:49:17.541102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.539340 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-data" pod "bc946897-4a76-4a95-b7ed-c9402f9a4d42" (UID: "bc946897-4a76-4a95-b7ed-c9402f9a4d42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.541102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.540291 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfe9d9e-818f-4909-a48e-555b42174d32-istiod-ca-cert" pod "5dfe9d9e-818f-4909-a48e-555b42174d32" (UID: "5dfe9d9e-818f-4909-a48e-555b42174d32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:49:17.541102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.540763 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-socket\") pod \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " Oct 30 23:49:17.541102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.540961 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-socket" pod "bc946897-4a76-4a95-b7ed-c9402f9a4d42" (UID: "bc946897-4a76-4a95-b7ed-c9402f9a4d42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.541102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.541038 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-credential-socket\") pod \"5dfe9d9e-818f-4909-a48e-555b42174d32\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " Oct 30 23:49:17.542133 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.541078 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-socket\") pod \"5dfe9d9e-818f-4909-a48e-555b42174d32\" (UID: \"5dfe9d9e-818f-4909-a48e-555b42174d32\") " Oct 30 23:49:17.542133 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.541273 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-envoy\") pod \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " Oct 30 23:49:17.542133 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.541432 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-podinfo\") pod \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\" (UID: \"bc946897-4a76-4a95-b7ed-c9402f9a4d42\") " Oct 30 23:49:17.543887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.543376 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-credential-socket" pod "5dfe9d9e-818f-4909-a48e-555b42174d32" (UID: "5dfe9d9e-818f-4909-a48e-555b42174d32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.543887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.543690 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-token" pod "bc946897-4a76-4a95-b7ed-c9402f9a4d42" (UID: "bc946897-4a76-4a95-b7ed-c9402f9a4d42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:17.543995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.543896 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.543995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.543913 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.543995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.543929 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf4s2\" (UniqueName: \"kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-kube-api-access-sf4s2\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.543995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.543943 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.543995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.543956 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.543995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.543970 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.543995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.543982 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.543995 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.543996 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe9d9e-818f-4909-a48e-555b42174d32-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.544934 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.544009 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792ae9e3-7af7-4767-900f-409a34cc02ac-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.544934 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.544174 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-socket" pod "5dfe9d9e-818f-4909-a48e-555b42174d32" (UID: "5dfe9d9e-818f-4909-a48e-555b42174d32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.544934 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.544848 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istiod-ca-cert" pod "bc946897-4a76-4a95-b7ed-c9402f9a4d42" (UID: "bc946897-4a76-4a95-b7ed-c9402f9a4d42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:49:17.545456 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.545305 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-credential-socket" pod "bc946897-4a76-4a95-b7ed-c9402f9a4d42" (UID: "bc946897-4a76-4a95-b7ed-c9402f9a4d42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.545831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.545567 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-data" pod "5dfe9d9e-818f-4909-a48e-555b42174d32" (UID: "5dfe9d9e-818f-4909-a48e-555b42174d32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.545831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.545607 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-certs" pod "5dfe9d9e-818f-4909-a48e-555b42174d32" (UID: "5dfe9d9e-818f-4909-a48e-555b42174d32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.548277 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.548041 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-podinfo" pod "bc946897-4a76-4a95-b7ed-c9402f9a4d42" (UID: "bc946897-4a76-4a95-b7ed-c9402f9a4d42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:49:17.548277 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.548233 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-certs" pod "bc946897-4a76-4a95-b7ed-c9402f9a4d42" (UID: "bc946897-4a76-4a95-b7ed-c9402f9a4d42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.550601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.549605 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-envoy" pod "5dfe9d9e-818f-4909-a48e-555b42174d32" (UID: "5dfe9d9e-818f-4909-a48e-555b42174d32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.550601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.549652 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-token" pod "5dfe9d9e-818f-4909-a48e-555b42174d32" (UID: "5dfe9d9e-818f-4909-a48e-555b42174d32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:17.553022 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.552979 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-kube-api-access-bjfjp" pod "bc946897-4a76-4a95-b7ed-c9402f9a4d42" (UID: "bc946897-4a76-4a95-b7ed-c9402f9a4d42"). InnerVolumeSpecName "kube-api-access-bjfjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:17.560532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.560454 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-envoy" pod "bc946897-4a76-4a95-b7ed-c9402f9a4d42" (UID: "bc946897-4a76-4a95-b7ed-c9402f9a4d42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.566867 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.566785 7570 scope.go:122] "RemoveContainer" containerID="dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272" Oct 30 23:49:17.569229 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:49:17.567614 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272\": container with ID starting with dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272 not found: ID does not exist" containerID="dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272" Oct 30 23:49:17.569229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.567652 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272"} err="failed to get container status \"dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272\": rpc error: code = NotFound desc = could not find container \"dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272\": container with ID starting with dd04976893f35ac70f2267e96f1d2b90fff762065ef8e4da9273516c9c609272 not found: ID does not exist" Oct 30 23:49:17.569229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.567675 7570 scope.go:122] "RemoveContainer" containerID="162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1" Oct 30 23:49:17.571742 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.571720 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl"] Oct 30 23:49:17.572933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.572871 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-kube-api-access-mdnh9" pod "5dfe9d9e-818f-4909-a48e-555b42174d32" (UID: "5dfe9d9e-818f-4909-a48e-555b42174d32"). InnerVolumeSpecName "kube-api-access-mdnh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:17.577333 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.577295 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-podinfo" pod "5dfe9d9e-818f-4909-a48e-555b42174d32" (UID: "5dfe9d9e-818f-4909-a48e-555b42174d32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:49:17.579410 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.579387 7570 scope.go:122] "RemoveContainer" containerID="162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1" Oct 30 23:49:17.579676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.579656 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-vmhzl"] Oct 30 23:49:17.579830 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:49:17.579806 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1\": container with ID starting with 162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1 not found: ID does not exist" containerID="162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1" Oct 30 23:49:17.579882 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.579842 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1"} err="failed to get container status \"162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1\": rpc error: code = NotFound desc = could not find container \"162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1\": container with ID starting with 162dbe4fe1218a42227f56b95be6eb74295a372e8d10b35316d814f3f82f71b1 not found: ID does not exist" Oct 30 23:49:17.579882 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.579867 7570 scope.go:122] "RemoveContainer" containerID="7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f" Oct 30 23:49:17.593577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.593543 7570 scope.go:122] "RemoveContainer" containerID="7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f" Oct 30 23:49:17.593909 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:49:17.593886 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f\": container with ID starting with 7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f not found: ID does not exist" containerID="7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f" Oct 30 23:49:17.594008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.593921 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f"} err="failed to get container status \"7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f\": rpc error: code = NotFound desc = could not find container \"7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f\": container with ID starting with 7e4bd327be9c8f8ab61cd2dc575ae1b515e4391d1c7c9c6692b435ee247e583f not found: ID does not exist" Oct 30 23:49:17.648889 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.648861 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.649114 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.649085 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdnh9\" (UniqueName: \"kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-kube-api-access-mdnh9\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.649114 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.649117 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.649308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.649149 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjfjp\" (UniqueName: \"kubernetes.io/projected/bc946897-4a76-4a95-b7ed-c9402f9a4d42-kube-api-access-bjfjp\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.649308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.649165 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.649308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.649180 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.649308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.649194 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.649308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.649207 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.649308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.649223 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.649308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.649236 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.649308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.649250 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe9d9e-818f-4909-a48e-555b42174d32-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.649308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.649265 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.649308 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.649279 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc946897-4a76-4a95-b7ed-c9402f9a4d42-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.702794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.702762 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:49:17.749955 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.749921 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-certs\") pod \"0c029726-9be9-4ac8-a268-8761a4499b87\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " Oct 30 23:49:17.750165 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.750095 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-istio-token\" (UniqueName: \"kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-istio-token\") pod \"0c029726-9be9-4ac8-a268-8761a4499b87\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " Oct 30 23:49:17.750255 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.750164 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-envoy\") pod \"0c029726-9be9-4ac8-a268-8761a4499b87\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " Oct 30 23:49:17.750255 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.750219 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c029726-9be9-4ac8-a268-8761a4499b87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c029726-9be9-4ac8-a268-8761a4499b87-istiod-ca-cert\") pod \"0c029726-9be9-4ac8-a268-8761a4499b87\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " Oct 30 23:49:17.750366 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.750266 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-credential-socket\") pod \"0c029726-9be9-4ac8-a268-8761a4499b87\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " Oct 30 23:49:17.750553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.750425 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c029726-9be9-4ac8-a268-8761a4499b87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c029726-9be9-4ac8-a268-8761a4499b87-istio-podinfo\") pod \"0c029726-9be9-4ac8-a268-8761a4499b87\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " Oct 30 23:49:17.750553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.750506 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-data\") pod \"0c029726-9be9-4ac8-a268-8761a4499b87\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " Oct 30 23:49:17.750744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.750566 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-socket\") pod \"0c029726-9be9-4ac8-a268-8761a4499b87\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " Oct 30 23:49:17.750744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.750653 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-kube-api-access-kzpd4\" (UniqueName: \"kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-kube-api-access-kzpd4\") pod \"0c029726-9be9-4ac8-a268-8761a4499b87\" (UID: \"0c029726-9be9-4ac8-a268-8761a4499b87\") " Oct 30 23:49:17.752878 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.751923 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-credential-socket" pod "0c029726-9be9-4ac8-a268-8761a4499b87" (UID: "0c029726-9be9-4ac8-a268-8761a4499b87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.752878 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.752510 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-certs" pod "0c029726-9be9-4ac8-a268-8761a4499b87" (UID: "0c029726-9be9-4ac8-a268-8761a4499b87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.752878 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.752606 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-socket" pod "0c029726-9be9-4ac8-a268-8761a4499b87" (UID: "0c029726-9be9-4ac8-a268-8761a4499b87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.753575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.753545 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-data" pod "0c029726-9be9-4ac8-a268-8761a4499b87" (UID: "0c029726-9be9-4ac8-a268-8761a4499b87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.753871 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.753856 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c029726-9be9-4ac8-a268-8761a4499b87-istiod-ca-cert" pod "0c029726-9be9-4ac8-a268-8761a4499b87" (UID: "0c029726-9be9-4ac8-a268-8761a4499b87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:49:17.757119 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.757088 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-istio-token" pod "0c029726-9be9-4ac8-a268-8761a4499b87" (UID: "0c029726-9be9-4ac8-a268-8761a4499b87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:17.757516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.757454 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-kube-api-access-kzpd4" pod "0c029726-9be9-4ac8-a268-8761a4499b87" (UID: "0c029726-9be9-4ac8-a268-8761a4499b87"). InnerVolumeSpecName "kube-api-access-kzpd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:17.757831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.757806 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-envoy" pod "0c029726-9be9-4ac8-a268-8761a4499b87" (UID: "0c029726-9be9-4ac8-a268-8761a4499b87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:17.760411 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.760384 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c029726-9be9-4ac8-a268-8761a4499b87-istio-podinfo" pod "0c029726-9be9-4ac8-a268-8761a4499b87" (UID: "0c029726-9be9-4ac8-a268-8761a4499b87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:49:17.836443 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.836409 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5"] Oct 30 23:49:17.840061 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.840036 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-cpbm5"] Oct 30 23:49:17.852902 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.852883 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzpd4\" (UniqueName: \"kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-kube-api-access-kzpd4\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.853008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.852905 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.853008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.852915 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c029726-9be9-4ac8-a268-8761a4499b87-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.853008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.852924 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.853008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.852934 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c029726-9be9-4ac8-a268-8761a4499b87-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.853008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.852942 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.853008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.852951 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c029726-9be9-4ac8-a268-8761a4499b87-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.853008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.852958 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.853008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.852966 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c029726-9be9-4ac8-a268-8761a4499b87-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:17.853819 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.853728 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24"] Oct 30 23:49:17.858064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:17.858043 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-clc24"] Oct 30 23:49:18.543254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:18.543222 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" event={"ID":"0c029726-9be9-4ac8-a268-8761a4499b87","Type":"ContainerDied","Data":"a2ee79c530fb41030342628f17c9a61da21420ecaeb57108dbfd16dbf5e90210"} Oct 30 23:49:18.543781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:18.543283 7570 scope.go:122] "RemoveContainer" containerID="73e9a637495396add78640f3ed8d7c33b22555c9bb55f7223f94a09d6628bc94" Oct 30 23:49:18.543781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:18.543308 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6" Oct 30 23:49:18.571935 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:18.571909 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6"] Oct 30 23:49:18.582522 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:18.582494 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-g9fw6"] Oct 30 23:49:18.603848 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:18.603822 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c029726-9be9-4ac8-a268-8761a4499b87" path="/var/lib/kubelet/pods/0c029726-9be9-4ac8-a268-8761a4499b87/volumes" Oct 30 23:49:18.604406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:18.604391 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" path="/var/lib/kubelet/pods/5dfe9d9e-818f-4909-a48e-555b42174d32/volumes" Oct 30 23:49:18.604955 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:18.604940 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="792ae9e3-7af7-4767-900f-409a34cc02ac" path="/var/lib/kubelet/pods/792ae9e3-7af7-4767-900f-409a34cc02ac/volumes" Oct 30 23:49:18.605764 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:18.605749 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" path="/var/lib/kubelet/pods/bc946897-4a76-4a95-b7ed-c9402f9a4d42/volumes" Oct 30 23:49:25.362254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.362216 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx"] Oct 30 23:49:25.363031 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.363014 7570 cpu_manager.go:414] "containerMap: removing container" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerName="istio-proxy" Oct 30 23:49:25.363112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.363032 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerName="istio-proxy" Oct 30 23:49:25.363112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.363045 7570 cpu_manager.go:414] "containerMap: removing container" podUID="792ae9e3-7af7-4767-900f-409a34cc02ac" containerName="istio-proxy" Oct 30 23:49:25.363112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.363050 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="792ae9e3-7af7-4767-900f-409a34cc02ac" containerName="istio-proxy" Oct 30 23:49:25.363112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.363064 7570 cpu_manager.go:414] "containerMap: removing container" podUID="0c029726-9be9-4ac8-a268-8761a4499b87" containerName="istio-proxy" Oct 30 23:49:25.363112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.363069 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c029726-9be9-4ac8-a268-8761a4499b87" containerName="istio-proxy" Oct 30 23:49:25.363112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.363076 7570 cpu_manager.go:414] "containerMap: removing container" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" Oct 30 23:49:25.363112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.363080 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" Oct 30 23:49:25.363587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.363216 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="792ae9e3-7af7-4767-900f-409a34cc02ac" containerName="istio-proxy" Oct 30 23:49:25.363587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.363226 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c029726-9be9-4ac8-a268-8761a4499b87" containerName="istio-proxy" Oct 30 23:49:25.363587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.363235 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc946897-4a76-4a95-b7ed-c9402f9a4d42" containerName="istio-proxy" Oct 30 23:49:25.363587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.363251 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dfe9d9e-818f-4909-a48e-555b42174d32" containerName="istio-proxy" Oct 30 23:49:25.369576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.369552 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:49:25.378760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.378728 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7fc6e9\"" Oct 30 23:49:25.380562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.380536 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx"] Oct 30 23:49:25.401252 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.401214 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt"] Oct 30 23:49:25.407788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.407753 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:49:25.422596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.422564 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt"] Oct 30 23:49:25.478355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.475981 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8fwp\" (UniqueName: \"kubernetes.io/projected/1d0b4116-23a4-4074-8dfb-93b7bbf53681-kube-api-access-n8fwp\") pod \"payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt\" (UID: \"1d0b4116-23a4-4074-8dfb-93b7bbf53681\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:49:25.478355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.476069 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0b4116-23a4-4074-8dfb-93b7bbf53681-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt\" (UID: \"1d0b4116-23a4-4074-8dfb-93b7bbf53681\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:49:25.478355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.476341 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08022df4-0b83-43af-9b14-b85ab9f83be3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx\" (UID: \"08022df4-0b83-43af-9b14-b85ab9f83be3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:49:25.478355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.476511 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jmmd\" (UniqueName: \"kubernetes.io/projected/08022df4-0b83-43af-9b14-b85ab9f83be3-kube-api-access-2jmmd\") pod \"payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx\" (UID: \"08022df4-0b83-43af-9b14-b85ab9f83be3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:49:25.503684 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.502832 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7fc6e9-dockercfg-85m2r\"" Oct 30 23:49:25.587488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.583904 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8fwp\" (UniqueName: \"kubernetes.io/projected/1d0b4116-23a4-4074-8dfb-93b7bbf53681-kube-api-access-n8fwp\") pod \"payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt\" (UID: \"1d0b4116-23a4-4074-8dfb-93b7bbf53681\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:49:25.587488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.584001 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0b4116-23a4-4074-8dfb-93b7bbf53681-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt\" (UID: \"1d0b4116-23a4-4074-8dfb-93b7bbf53681\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:49:25.587488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.584108 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08022df4-0b83-43af-9b14-b85ab9f83be3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx\" (UID: \"08022df4-0b83-43af-9b14-b85ab9f83be3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:49:25.587488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.584234 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmmd\" (UniqueName: \"kubernetes.io/projected/08022df4-0b83-43af-9b14-b85ab9f83be3-kube-api-access-2jmmd\") pod \"payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx\" (UID: \"08022df4-0b83-43af-9b14-b85ab9f83be3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:49:25.587488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.585674 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0b4116-23a4-4074-8dfb-93b7bbf53681-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt\" (UID: \"1d0b4116-23a4-4074-8dfb-93b7bbf53681\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:49:25.587488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.585837 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08022df4-0b83-43af-9b14-b85ab9f83be3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx\" (UID: \"08022df4-0b83-43af-9b14-b85ab9f83be3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:49:25.673536 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.670797 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8fwp\" (UniqueName: \"kubernetes.io/projected/1d0b4116-23a4-4074-8dfb-93b7bbf53681-kube-api-access-n8fwp\") pod \"payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt\" (UID: \"1d0b4116-23a4-4074-8dfb-93b7bbf53681\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:49:25.678555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.677200 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jmmd\" (UniqueName: \"kubernetes.io/projected/08022df4-0b83-43af-9b14-b85ab9f83be3-kube-api-access-2jmmd\") pod \"payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx\" (UID: \"08022df4-0b83-43af-9b14-b85ab9f83be3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:49:25.699535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.692311 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:49:25.758489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:25.734017 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:49:26.282964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.282929 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx"] Oct 30 23:49:26.283173 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:49:26.283147 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08022df4_0b83_43af_9b14_b85ab9f83be3.slice/crio-d0d37036f57057b511292cea8c380493d877015b8b3d6c6952b4d755c8f0871c WatchSource:0}: Error finding container d0d37036f57057b511292cea8c380493d877015b8b3d6c6952b4d755c8f0871c: Status 404 returned error can't find the container with id d0d37036f57057b511292cea8c380493d877015b8b3d6c6952b4d755c8f0871c Oct 30 23:49:26.391113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.391083 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt"] Oct 30 23:49:26.392569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:49:26.392539 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0b4116_23a4_4074_8dfb_93b7bbf53681.slice/crio-c4f694c6311c81e7e237465327a6f8e87cfbce8b7e45e59e5e3e2cf097fc39bb WatchSource:0}: Error finding container c4f694c6311c81e7e237465327a6f8e87cfbce8b7e45e59e5e3e2cf097fc39bb: Status 404 returned error can't find the container with id c4f694c6311c81e7e237465327a6f8e87cfbce8b7e45e59e5e3e2cf097fc39bb Oct 30 23:49:26.681377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.681338 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" event={"ID":"08022df4-0b83-43af-9b14-b85ab9f83be3","Type":"ContainerStarted","Data":"5586221ccce00e8910c448c68c374e36fb26c7ec0d90934b6f10d1f3ed13ceed"} Oct 30 23:49:26.681605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.681412 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:49:26.681605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.681441 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" event={"ID":"08022df4-0b83-43af-9b14-b85ab9f83be3","Type":"ContainerStarted","Data":"d0d37036f57057b511292cea8c380493d877015b8b3d6c6952b4d755c8f0871c"} Oct 30 23:49:26.683139 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.683109 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" event={"ID":"1d0b4116-23a4-4074-8dfb-93b7bbf53681","Type":"ContainerStarted","Data":"6d0a6bd4c7aa80c7f055c16f2c9eec203d6315d8f85d57501dd965260c5b3285"} Oct 30 23:49:26.683249 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.683163 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" event={"ID":"1d0b4116-23a4-4074-8dfb-93b7bbf53681","Type":"ContainerStarted","Data":"c4f694c6311c81e7e237465327a6f8e87cfbce8b7e45e59e5e3e2cf097fc39bb"} Oct 30 23:49:26.683249 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.683186 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:49:26.703724 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.703662 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" podStartSLOduration=1.703648354 podStartE2EDuration="1.703648354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:26.701548531 +0000 UTC m=+1778.220549622" watchObservedRunningTime="2026-10-30 23:49:26.703648354 +0000 UTC m=+1778.222649444" Oct 30 23:49:26.724812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.724765 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" podStartSLOduration=1.724751699 podStartE2EDuration="1.724751699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:26.722347128 +0000 UTC m=+1778.241348215" watchObservedRunningTime="2026-10-30 23:49:26.724751699 +0000 UTC m=+1778.243752788" Oct 30 23:49:26.905770 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.905731 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs"] Oct 30 23:49:26.906035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.906005 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" podUID="e618c702-d336-43d9-8857-430b4fa5b8ce" containerName="istio-proxy" containerID="cri-o://e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf" gracePeriod=30 Oct 30 23:49:26.910544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.910520 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n"] Oct 30 23:49:26.910749 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:26.910725 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" containerID="cri-o://37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb" gracePeriod=30 Oct 30 23:49:29.419874 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:49:29.419837 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:49:30.256784 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.256747 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp"] Oct 30 23:49:30.263492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.263453 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.271791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.271748 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a63288-openshift-default-dockercfg-tdk54\"" Oct 30 23:49:30.277210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.277073 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp"] Oct 30 23:49:30.371486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.370563 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-podinfo\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.371486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.370666 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-credential-socket\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.371486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.370727 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-socket\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.371486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.370762 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-certs\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.371486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.370831 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-data\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.371486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.370854 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-envoy\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.371486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.370913 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-token\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.371486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.370964 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istiod-ca-cert\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.371486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.371046 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwsfc\" (UniqueName: \"kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-kube-api-access-cwsfc\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.472592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.472553 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-podinfo\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.475212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.475185 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-credential-socket\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.475450 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.475430 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-socket\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.475644 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.475627 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-certs\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.481839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.481810 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-data\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.482022 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.482006 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-envoy\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.482564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.482541 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-token\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.482818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.482797 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istiod-ca-cert\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.483102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.483076 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsfc\" (UniqueName: \"kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-kube-api-access-cwsfc\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.483711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.475862 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-credential-socket\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.484182 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.484161 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-data\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.484316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.476093 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-socket\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.493766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.476359 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-certs\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.493766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.490407 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istiod-ca-cert\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.507613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.506443 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-podinfo\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.508906 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.508877 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwsfc\" (UniqueName: \"kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-kube-api-access-cwsfc\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.512154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.512133 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-token\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.537444 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.537407 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-envoy\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.591496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.586848 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:30.921826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.921798 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp"] Oct 30 23:49:30.923546 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:49:30.923517 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3856a78_100e_48e0_aa1e_9a04bb10c99d.slice/crio-0233294017b589cf09a602503e5aab60a552d54bde063fde208d73ed4620ade2 WatchSource:0}: Error finding container 0233294017b589cf09a602503e5aab60a552d54bde063fde208d73ed4620ade2: Status 404 returned error can't find the container with id 0233294017b589cf09a602503e5aab60a552d54bde063fde208d73ed4620ade2 Oct 30 23:49:30.926768 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.926729 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:49:30.926863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.926818 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:49:30.926903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:30.926867 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:49:31.749169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:31.749126 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" event={"ID":"f3856a78-100e-48e0-aa1e-9a04bb10c99d","Type":"ContainerStarted","Data":"5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870"} Oct 30 23:49:31.749678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:31.749201 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" event={"ID":"f3856a78-100e-48e0-aa1e-9a04bb10c99d","Type":"ContainerStarted","Data":"0233294017b589cf09a602503e5aab60a552d54bde063fde208d73ed4620ade2"} Oct 30 23:49:31.783287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:31.783232 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" podStartSLOduration=1.783217979 podStartE2EDuration="1.783217979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:31.778422397 +0000 UTC m=+1783.297423488" watchObservedRunningTime="2026-10-30 23:49:31.783217979 +0000 UTC m=+1783.302219068" Oct 30 23:49:32.253798 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.253763 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:49:32.270445 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.270421 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:49:32.439448 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439413 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-envoy\") pod \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " Oct 30 23:49:32.439448 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439456 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-podinfo\") pod \"e618c702-d336-43d9-8857-430b4fa5b8ce\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " Oct 30 23:49:32.439839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439564 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-credential-socket\") pod \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " Oct 30 23:49:32.439839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439586 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-certs\") pod \"e618c702-d336-43d9-8857-430b4fa5b8ce\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " Oct 30 23:49:32.439839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439625 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-kube-api-access-q4vx7\" (UniqueName: \"kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-kube-api-access-q4vx7\") pod \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " Oct 30 23:49:32.439839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439655 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-envoy\") pod \"e618c702-d336-43d9-8857-430b4fa5b8ce\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " Oct 30 23:49:32.439839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439705 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istiod-ca-cert\") pod \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " Oct 30 23:49:32.439839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439755 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-data\") pod \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " Oct 30 23:49:32.439839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439796 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-token\" (UniqueName: \"kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-token\") pod \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " Oct 30 23:49:32.439839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439827 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-credential-socket" pod "d42ff644-b417-4c6e-9ecf-aea24c6233d9" (UID: "d42ff644-b417-4c6e-9ecf-aea24c6233d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:32.440278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439855 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e618c702-d336-43d9-8857-430b4fa5b8ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e618c702-d336-43d9-8857-430b4fa5b8ce-istiod-ca-cert\") pod \"e618c702-d336-43d9-8857-430b4fa5b8ce\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " Oct 30 23:49:32.440278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439896 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-socket\") pod \"e618c702-d336-43d9-8857-430b4fa5b8ce\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " Oct 30 23:49:32.440278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439945 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-certs\") pod \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " Oct 30 23:49:32.440278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.439955 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-certs" pod "e618c702-d336-43d9-8857-430b4fa5b8ce" (UID: "e618c702-d336-43d9-8857-430b4fa5b8ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:32.440278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.440024 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-socket\") pod \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " Oct 30 23:49:32.440278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.440072 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-podinfo\") pod \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\" (UID: \"d42ff644-b417-4c6e-9ecf-aea24c6233d9\") " Oct 30 23:49:32.440278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.440126 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istiod-ca-cert" pod "d42ff644-b417-4c6e-9ecf-aea24c6233d9" (UID: "d42ff644-b417-4c6e-9ecf-aea24c6233d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:49:32.440278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.440239 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-kube-api-access-7kgr2\" (UniqueName: \"kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-kube-api-access-7kgr2\") pod \"e618c702-d336-43d9-8857-430b4fa5b8ce\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " Oct 30 23:49:32.440727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.440285 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e618c702-d336-43d9-8857-430b4fa5b8ce-istiod-ca-cert" pod "e618c702-d336-43d9-8857-430b4fa5b8ce" (UID: "e618c702-d336-43d9-8857-430b4fa5b8ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:49:32.440727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.440362 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-token\" (UniqueName: \"kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-token\") pod \"e618c702-d336-43d9-8857-430b4fa5b8ce\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " Oct 30 23:49:32.440727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.440439 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-credential-socket\") pod \"e618c702-d336-43d9-8857-430b4fa5b8ce\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " Oct 30 23:49:32.440727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.440544 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-data\") pod \"e618c702-d336-43d9-8857-430b4fa5b8ce\" (UID: \"e618c702-d336-43d9-8857-430b4fa5b8ce\") " Oct 30 23:49:32.440727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.440590 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-data" pod "d42ff644-b417-4c6e-9ecf-aea24c6233d9" (UID: "d42ff644-b417-4c6e-9ecf-aea24c6233d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:32.440727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.440703 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-socket" pod "d42ff644-b417-4c6e-9ecf-aea24c6233d9" (UID: "d42ff644-b417-4c6e-9ecf-aea24c6233d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:32.441051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.440910 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-socket" pod "e618c702-d336-43d9-8857-430b4fa5b8ce" (UID: "e618c702-d336-43d9-8857-430b4fa5b8ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:32.441118 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441091 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-certs" pod "d42ff644-b417-4c6e-9ecf-aea24c6233d9" (UID: "d42ff644-b417-4c6e-9ecf-aea24c6233d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:32.441389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441256 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-credential-socket" pod "e618c702-d336-43d9-8857-430b4fa5b8ce" (UID: "e618c702-d336-43d9-8857-430b4fa5b8ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:32.441503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441483 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-data" pod "e618c702-d336-43d9-8857-430b4fa5b8ce" (UID: "e618c702-d336-43d9-8857-430b4fa5b8ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:32.441755 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441731 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.441853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441756 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.441853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441771 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.441853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441786 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.441853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441801 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.441853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441815 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.441853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441830 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.441853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441844 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e618c702-d336-43d9-8857-430b4fa5b8ce-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.442253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441859 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.442253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.441872 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.442776 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.442556 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-envoy" pod "d42ff644-b417-4c6e-9ecf-aea24c6233d9" (UID: "d42ff644-b417-4c6e-9ecf-aea24c6233d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:32.442776 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.442738 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-token" pod "d42ff644-b417-4c6e-9ecf-aea24c6233d9" (UID: "d42ff644-b417-4c6e-9ecf-aea24c6233d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:32.443019 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.442974 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-envoy" pod "e618c702-d336-43d9-8857-430b4fa5b8ce" (UID: "e618c702-d336-43d9-8857-430b4fa5b8ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:32.443019 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.442990 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-podinfo" pod "e618c702-d336-43d9-8857-430b4fa5b8ce" (UID: "e618c702-d336-43d9-8857-430b4fa5b8ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:49:32.443223 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.443200 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-kube-api-access-7kgr2" pod "e618c702-d336-43d9-8857-430b4fa5b8ce" (UID: "e618c702-d336-43d9-8857-430b4fa5b8ce"). InnerVolumeSpecName "kube-api-access-7kgr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:32.444371 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.444347 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-kube-api-access-q4vx7" pod "d42ff644-b417-4c6e-9ecf-aea24c6233d9" (UID: "d42ff644-b417-4c6e-9ecf-aea24c6233d9"). InnerVolumeSpecName "kube-api-access-q4vx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:32.444570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.444543 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-token" pod "e618c702-d336-43d9-8857-430b4fa5b8ce" (UID: "e618c702-d336-43d9-8857-430b4fa5b8ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:32.444658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.444585 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-podinfo" pod "d42ff644-b417-4c6e-9ecf-aea24c6233d9" (UID: "d42ff644-b417-4c6e-9ecf-aea24c6233d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:49:32.544340 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.544250 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.544340 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.544284 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.544340 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.544298 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kgr2\" (UniqueName: \"kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-kube-api-access-7kgr2\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.544340 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.544310 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.544340 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.544323 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42ff644-b417-4c6e-9ecf-aea24c6233d9-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.544340 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.544335 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.544731 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.544348 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4vx7\" (UniqueName: \"kubernetes.io/projected/d42ff644-b417-4c6e-9ecf-aea24c6233d9-kube-api-access-q4vx7\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.544731 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.544360 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e618c702-d336-43d9-8857-430b4fa5b8ce-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:32.589101 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.589065 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:32.606179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.606152 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:32.758733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.758698 7570 generic.go:356] "Generic (PLEG): container finished" podID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerID="37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb" exitCode=0 Oct 30 23:49:32.759281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.758755 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" Oct 30 23:49:32.759281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.758772 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" event={"ID":"d42ff644-b417-4c6e-9ecf-aea24c6233d9","Type":"ContainerDied","Data":"37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb"} Oct 30 23:49:32.759281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.758844 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n" event={"ID":"d42ff644-b417-4c6e-9ecf-aea24c6233d9","Type":"ContainerDied","Data":"7e3d2667ce714f6c56c54e0221686694e966ba8f20287ceeab5c26ca55ad96c0"} Oct 30 23:49:32.759281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.758879 7570 scope.go:122] "RemoveContainer" containerID="37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb" Oct 30 23:49:32.760608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.760404 7570 generic.go:356] "Generic (PLEG): container finished" podID="e618c702-d336-43d9-8857-430b4fa5b8ce" containerID="e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf" exitCode=0 Oct 30 23:49:32.760608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.760478 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" Oct 30 23:49:32.760608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.760497 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" event={"ID":"e618c702-d336-43d9-8857-430b4fa5b8ce","Type":"ContainerDied","Data":"e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf"} Oct 30 23:49:32.760608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.760548 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs" event={"ID":"e618c702-d336-43d9-8857-430b4fa5b8ce","Type":"ContainerDied","Data":"bb833b9595500cffccb95ab2ef0970cf8cc9c5c250b6057db1fd00db7d665d61"} Oct 30 23:49:32.762143 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.760971 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:32.762377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.762357 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:49:32.772755 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.772736 7570 scope.go:122] "RemoveContainer" containerID="37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb" Oct 30 23:49:32.773043 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:49:32.773025 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb\": container with ID starting with 37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb not found: ID does not exist" containerID="37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb" Oct 30 23:49:32.773094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.773051 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb"} err="failed to get container status \"37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb\": rpc error: code = NotFound desc = could not find container \"37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb\": container with ID starting with 37923cc0d72ef2e58c2baf659ae91cfcb9b02ed020dbb3f8864381bec1a9f6cb not found: ID does not exist" Oct 30 23:49:32.773094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.773067 7570 scope.go:122] "RemoveContainer" containerID="e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf" Oct 30 23:49:32.784063 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.784041 7570 scope.go:122] "RemoveContainer" containerID="e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf" Oct 30 23:49:32.784349 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:49:32.784327 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf\": container with ID starting with e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf not found: ID does not exist" containerID="e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf" Oct 30 23:49:32.784426 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.784360 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf"} err="failed to get container status \"e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf\": rpc error: code = NotFound desc = could not find container \"e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf\": container with ID starting with e1626ff054934fa28fdbb5635ee9c3d313594af1407d80de69ac68be537aeddf not found: ID does not exist" Oct 30 23:49:32.823983 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.823902 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs"] Oct 30 23:49:32.838892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.838859 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-t5pfs"] Oct 30 23:49:32.864513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.864476 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n"] Oct 30 23:49:32.870118 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:32.870084 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-k497n"] Oct 30 23:49:34.604817 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:34.604786 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" path="/var/lib/kubelet/pods/d42ff644-b417-4c6e-9ecf-aea24c6233d9/volumes" Oct 30 23:49:34.605342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:34.605328 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e618c702-d336-43d9-8857-430b4fa5b8ce" path="/var/lib/kubelet/pods/e618c702-d336-43d9-8857-430b4fa5b8ce/volumes" Oct 30 23:49:35.039196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.039104 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf"] Oct 30 23:49:35.042696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.042672 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b487ee95-4jldf"] Oct 30 23:49:35.864029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.863998 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp"] Oct 30 23:49:35.864848 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.864831 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e618c702-d336-43d9-8857-430b4fa5b8ce" containerName="istio-proxy" Oct 30 23:49:35.864848 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.864848 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e618c702-d336-43d9-8857-430b4fa5b8ce" containerName="istio-proxy" Oct 30 23:49:35.865009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.864911 7570 cpu_manager.go:414] "containerMap: removing container" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" Oct 30 23:49:35.865009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.864916 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" Oct 30 23:49:35.865117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.865024 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42ff644-b417-4c6e-9ecf-aea24c6233d9" containerName="istio-proxy" Oct 30 23:49:35.865117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.865036 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e618c702-d336-43d9-8857-430b4fa5b8ce" containerName="istio-proxy" Oct 30 23:49:35.870299 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.870280 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:49:35.883905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.883550 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp"] Oct 30 23:49:35.884824 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.884801 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a63288\"" Oct 30 23:49:35.894587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.894559 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx"] Oct 30 23:49:35.900945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.900850 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:49:35.909994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.909972 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a63288-dockercfg-5m62n\"" Oct 30 23:49:35.917278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.915820 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp\" (UID: \"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:49:35.917278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.915898 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4bhr\" (UniqueName: \"kubernetes.io/projected/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-kube-api-access-w4bhr\") pod \"payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp\" (UID: \"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:49:35.921447 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:35.921139 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx"] Oct 30 23:49:36.074488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.073548 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6843e367-692b-41f0-a9c6-7704d732f009-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx\" (UID: \"6843e367-692b-41f0-a9c6-7704d732f009\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:49:36.074488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.073644 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp\" (UID: \"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:49:36.074488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.073719 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbmf8\" (UniqueName: \"kubernetes.io/projected/6843e367-692b-41f0-a9c6-7704d732f009-kube-api-access-lbmf8\") pod \"payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx\" (UID: \"6843e367-692b-41f0-a9c6-7704d732f009\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:49:36.074488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.073763 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bhr\" (UniqueName: \"kubernetes.io/projected/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-kube-api-access-w4bhr\") pod \"payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp\" (UID: \"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:49:36.095487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.075225 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp\" (UID: \"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:49:36.171285 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.171189 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bhr\" (UniqueName: \"kubernetes.io/projected/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-kube-api-access-w4bhr\") pod \"payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp\" (UID: \"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:49:36.182810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.182769 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbmf8\" (UniqueName: \"kubernetes.io/projected/6843e367-692b-41f0-a9c6-7704d732f009-kube-api-access-lbmf8\") pod \"payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx\" (UID: \"6843e367-692b-41f0-a9c6-7704d732f009\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:49:36.183441 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.183419 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6843e367-692b-41f0-a9c6-7704d732f009-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx\" (UID: \"6843e367-692b-41f0-a9c6-7704d732f009\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:49:36.184592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.184567 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6843e367-692b-41f0-a9c6-7704d732f009-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx\" (UID: \"6843e367-692b-41f0-a9c6-7704d732f009\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:49:36.189496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.189451 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:49:36.197892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.197865 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbmf8\" (UniqueName: \"kubernetes.io/projected/6843e367-692b-41f0-a9c6-7704d732f009-kube-api-access-lbmf8\") pod \"payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx\" (UID: \"6843e367-692b-41f0-a9c6-7704d732f009\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:49:36.230691 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.230663 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:49:36.717919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.716214 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231f6ea2-fcf2-416c-adf5-8b195cb0df01" path="/var/lib/kubelet/pods/231f6ea2-fcf2-416c-adf5-8b195cb0df01/volumes" Oct 30 23:49:36.723504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.721345 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp"] Oct 30 23:49:36.732250 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.732217 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx"] Oct 30 23:49:36.733848 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:49:36.733794 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6843e367_692b_41f0_a9c6_7704d732f009.slice/crio-9ed163a296ee9db20106e8dd23c424d0fcc7fbda9a675c853fd1d7b39783e95b WatchSource:0}: Error finding container 9ed163a296ee9db20106e8dd23c424d0fcc7fbda9a675c853fd1d7b39783e95b: Status 404 returned error can't find the container with id 9ed163a296ee9db20106e8dd23c424d0fcc7fbda9a675c853fd1d7b39783e95b Oct 30 23:49:36.811712 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.811670 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" event={"ID":"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d","Type":"ContainerStarted","Data":"7558b12fd5d8ff18a6d85e355482072e59d6decd7da49a152728deb9ad9d1150"} Oct 30 23:49:36.813180 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:36.813154 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" event={"ID":"6843e367-692b-41f0-a9c6-7704d732f009","Type":"ContainerStarted","Data":"9ed163a296ee9db20106e8dd23c424d0fcc7fbda9a675c853fd1d7b39783e95b"} Oct 30 23:49:37.694599 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:37.694570 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:49:37.695028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:37.694650 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:49:37.824843 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:37.824790 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" event={"ID":"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d","Type":"ContainerStarted","Data":"b790873757bf25b53bae157fa4d35b942dfd574013d0a1f057ee375f4e365537"} Oct 30 23:49:37.825032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:37.824864 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:49:37.828028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:37.827997 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" event={"ID":"6843e367-692b-41f0-a9c6-7704d732f009","Type":"ContainerStarted","Data":"14e282ade51dac8dc806a37d0545f5277b669f6986fb50c4a208641e5ab6f498"} Oct 30 23:49:37.828174 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:37.828086 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:49:37.848323 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:37.848265 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" podStartSLOduration=2.848248382 podStartE2EDuration="2.848248382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:37.84593297 +0000 UTC m=+1789.364934060" watchObservedRunningTime="2026-10-30 23:49:37.848248382 +0000 UTC m=+1789.367249473" Oct 30 23:49:37.877905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:37.877844 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" podStartSLOduration=2.877829604 podStartE2EDuration="2.877829604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:37.873169615 +0000 UTC m=+1789.392170705" watchObservedRunningTime="2026-10-30 23:49:37.877829604 +0000 UTC m=+1789.396830752" Oct 30 23:49:39.918587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:39.918551 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf"] Oct 30 23:49:39.924548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:39.924518 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:39.927846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:39.927817 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf"] Oct 30 23:49:39.934313 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:39.934277 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-23f75e-openshift-default-dockercfg-ngphj\"" Oct 30 23:49:40.017796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.012864 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-podinfo\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.017796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.012950 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-credential-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.017796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.012980 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94sb9\" (UniqueName: \"kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-kube-api-access-94sb9\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.017796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.013005 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f179a0a0-c464-4a8b-b6cc-04a21610a832-istiod-ca-cert\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.017796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.013113 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-certs\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.017796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.013179 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-envoy\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.017796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.013208 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-token\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.017796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.013387 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.017796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.013446 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-data\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.115501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.115449 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-podinfo\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.115807 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.115783 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-credential-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.115933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.115918 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94sb9\" (UniqueName: \"kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-kube-api-access-94sb9\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.116101 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.116027 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f179a0a0-c464-4a8b-b6cc-04a21610a832-istiod-ca-cert\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.122490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.116312 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-certs\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.122750 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.122725 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-envoy\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.122893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.122878 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-token\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.123196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.123176 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.123368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.123350 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-data\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.131490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.123965 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-data\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.131731 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.118286 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-credential-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.154492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.119731 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f179a0a0-c464-4a8b-b6cc-04a21610a832-istiod-ca-cert\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.154492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.119965 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-certs\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.154492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.142923 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.155143 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.155115 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-envoy\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.161914 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.159136 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94sb9\" (UniqueName: \"kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-kube-api-access-94sb9\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.184534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.172030 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-podinfo\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.184534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.184276 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-token\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.252492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.251605 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:40.720819 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.720780 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf"] Oct 30 23:49:40.730193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.730138 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:49:40.730334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.730296 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:49:40.730410 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.730389 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:49:40.856415 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.856369 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" event={"ID":"f179a0a0-c464-4a8b-b6cc-04a21610a832","Type":"ContainerStarted","Data":"4b453b82afffbfbaa0b1404213c2ba27f6c3443fb37a1de5aad05d9c30a3bc35"} Oct 30 23:49:40.907488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:40.898186 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podStartSLOduration=1.898164744 podStartE2EDuration="1.898164744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:40.881354362 +0000 UTC m=+1792.400355462" watchObservedRunningTime="2026-10-30 23:49:40.898164744 +0000 UTC m=+1792.417165835" Oct 30 23:49:41.255155 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:41.255116 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:41.256588 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:41.256557 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 23:49:41.256730 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:41.256616 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 23:49:41.868695 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:41.868649 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" event={"ID":"f179a0a0-c464-4a8b-b6cc-04a21610a832","Type":"ContainerStarted","Data":"6240555c4a0f290fefac4ba05d496628c7c52e564af57f302251937420a0f092"} Oct 30 23:49:42.260355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:42.260269 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:42.364933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:42.364894 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2"] Oct 30 23:49:42.365128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:42.365098 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" containerID="cri-o://c6a4e1f7e0b4586b2ac7fe2223526016c1728d8d667ebecad7ed2c062d5eae65" gracePeriod=30 Oct 30 23:49:42.878061 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:42.878031 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:42.879154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:42.879128 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:49:44.237650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.237557 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx"] Oct 30 23:49:44.262924 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.262897 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:44.300526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.281555 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx"] Oct 30 23:49:44.337485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.333869 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7fc6e9-serving-cert\"" Oct 30 23:49:44.436898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.436866 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgzmg\" (UniqueName: \"kubernetes.io/projected/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-kube-api-access-jgzmg\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:44.437267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.437232 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:44.437422 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.437304 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-metrics-certs\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:44.437525 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.437433 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:44.538778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.538701 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:44.538778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.538758 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-metrics-certs\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:44.539024 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:49:44.538818 7570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-serving-cert: secret "maas-api-e2e-inf-a-7fc6e9-serving-cert" not found Oct 30 23:49:44.539024 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.538860 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:44.539024 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:49:44.538886 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls podName:cc2a6d9c-ec3e-438a-9d6b-db75fa30a903 nodeName:}" failed. No retries permitted until 2026-10-30 23:49:45.038866813 +0000 UTC m=+1796.557867898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls") pod "maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" (UID: "cc2a6d9c-ec3e-438a-9d6b-db75fa30a903") : secret "maas-api-e2e-inf-a-7fc6e9-serving-cert" not found Oct 30 23:49:44.539024 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.539009 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzmg\" (UniqueName: \"kubernetes.io/projected/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-kube-api-access-jgzmg\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:44.539581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.539558 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:44.542119 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.542087 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-metrics-certs\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:44.554712 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.554688 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgzmg\" (UniqueName: \"kubernetes.io/projected/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-kube-api-access-jgzmg\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:44.791871 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.791772 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:49:44.791871 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:44.791850 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:49:45.049384 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.049293 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:45.049567 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:49:45.049456 7570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-serving-cert: secret "maas-api-e2e-inf-a-7fc6e9-serving-cert" not found Oct 30 23:49:45.049644 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:49:45.049623 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls podName:cc2a6d9c-ec3e-438a-9d6b-db75fa30a903 nodeName:}" failed. No retries permitted until 2026-10-30 23:49:46.049604002 +0000 UTC m=+1797.568605085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls") pod "maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" (UID: "cc2a6d9c-ec3e-438a-9d6b-db75fa30a903") : secret "maas-api-e2e-inf-a-7fc6e9-serving-cert" not found Oct 30 23:49:45.543080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.543044 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll"] Oct 30 23:49:45.548035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.548014 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:49:45.554908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.554885 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-23f75e-dockercfg-w5fh9\"" Oct 30 23:49:45.554908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.554931 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-23f75e\"" Oct 30 23:49:45.557058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.557032 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74a48f3-2b68-424b-9489-33e22dab2f2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll\" (UID: \"a74a48f3-2b68-424b-9489-33e22dab2f2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:49:45.557209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.557174 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zms24\" (UniqueName: \"kubernetes.io/projected/a74a48f3-2b68-424b-9489-33e22dab2f2f-kube-api-access-zms24\") pod \"payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll\" (UID: \"a74a48f3-2b68-424b-9489-33e22dab2f2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:49:45.558427 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.558401 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll"] Oct 30 23:49:45.588277 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.588218 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8"] Oct 30 23:49:45.594907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.594779 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:49:45.612330 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.612295 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8"] Oct 30 23:49:45.658827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.658786 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74a48f3-2b68-424b-9489-33e22dab2f2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll\" (UID: \"a74a48f3-2b68-424b-9489-33e22dab2f2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:49:45.658827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.658850 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx4dn\" (UniqueName: \"kubernetes.io/projected/fb14f0ca-115b-4fe6-afcd-6338df872573-kube-api-access-fx4dn\") pod \"payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8\" (UID: \"fb14f0ca-115b-4fe6-afcd-6338df872573\") " pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:49:45.659123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.658911 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zms24\" (UniqueName: \"kubernetes.io/projected/a74a48f3-2b68-424b-9489-33e22dab2f2f-kube-api-access-zms24\") pod \"payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll\" (UID: \"a74a48f3-2b68-424b-9489-33e22dab2f2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:49:45.659123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.659048 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb14f0ca-115b-4fe6-afcd-6338df872573-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8\" (UID: \"fb14f0ca-115b-4fe6-afcd-6338df872573\") " pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:49:45.660211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.660184 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74a48f3-2b68-424b-9489-33e22dab2f2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll\" (UID: \"a74a48f3-2b68-424b-9489-33e22dab2f2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:49:45.673646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.673594 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zms24\" (UniqueName: \"kubernetes.io/projected/a74a48f3-2b68-424b-9489-33e22dab2f2f-kube-api-access-zms24\") pod \"payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll\" (UID: \"a74a48f3-2b68-424b-9489-33e22dab2f2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:49:45.773490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.773106 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb14f0ca-115b-4fe6-afcd-6338df872573-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8\" (UID: \"fb14f0ca-115b-4fe6-afcd-6338df872573\") " pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:49:45.773686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.773526 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx4dn\" (UniqueName: \"kubernetes.io/projected/fb14f0ca-115b-4fe6-afcd-6338df872573-kube-api-access-fx4dn\") pod \"payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8\" (UID: \"fb14f0ca-115b-4fe6-afcd-6338df872573\") " pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:49:45.781486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.775055 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb14f0ca-115b-4fe6-afcd-6338df872573-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8\" (UID: \"fb14f0ca-115b-4fe6-afcd-6338df872573\") " pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:49:45.812531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.810453 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx4dn\" (UniqueName: \"kubernetes.io/projected/fb14f0ca-115b-4fe6-afcd-6338df872573-kube-api-access-fx4dn\") pod \"payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8\" (UID: \"fb14f0ca-115b-4fe6-afcd-6338df872573\") " pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:49:45.887483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.871774 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:49:45.926338 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:45.925940 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:49:46.087075 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.086996 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:46.104386 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.104315 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls\") pod \"maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:46.154710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.154678 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:46.354569 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:49:46.354513 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74a48f3_2b68_424b_9489_33e22dab2f2f.slice/crio-a043f90e7035a4e386bc15823e6385c6a5447d6ef823be9492939d680a4f67cc WatchSource:0}: Error finding container a043f90e7035a4e386bc15823e6385c6a5447d6ef823be9492939d680a4f67cc: Status 404 returned error can't find the container with id a043f90e7035a4e386bc15823e6385c6a5447d6ef823be9492939d680a4f67cc Oct 30 23:49:46.359054 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.358681 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll"] Oct 30 23:49:46.362852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.362826 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8"] Oct 30 23:49:46.487791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.487754 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk"] Oct 30 23:49:46.528446 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.528411 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk"] Oct 30 23:49:46.528671 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.528529 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.533010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.532384 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx"] Oct 30 23:49:46.535101 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:49:46.535069 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2a6d9c_ec3e_438a_9d6b_db75fa30a903.slice/crio-f3eb1ef19d45548ba59c05974de3f515521a241a1d6c23b0c274cea205ac46df WatchSource:0}: Error finding container f3eb1ef19d45548ba59c05974de3f515521a241a1d6c23b0c274cea205ac46df: Status 404 returned error can't find the container with id f3eb1ef19d45548ba59c05974de3f515521a241a1d6c23b0c274cea205ac46df Oct 30 23:49:46.538402 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.537549 7570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:49:46.541836 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.541804 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-82d449-openshift-default-dockercfg-mhmr7\"" Oct 30 23:49:46.804610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.803651 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af49f007-5f21-43cc-b4f3-d78e31b45100-istiod-ca-cert\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.804610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.803724 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-data\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.804610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.803782 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-podinfo\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.804610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.803826 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r6db\" (UniqueName: \"kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-kube-api-access-8r6db\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.804610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.803895 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-envoy\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.804610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.803934 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-socket\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.804610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.804021 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-credential-socket\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.804610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.804184 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-certs\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.804610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.804271 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-token\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.905646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.905606 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-token\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.906132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.906095 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af49f007-5f21-43cc-b4f3-d78e31b45100-istiod-ca-cert\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.906245 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.906153 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-data\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.906245 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.906197 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-podinfo\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.906245 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.906227 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r6db\" (UniqueName: \"kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-kube-api-access-8r6db\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.906416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.906276 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-envoy\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.906416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.906309 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-socket\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.906416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.906381 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-credential-socket\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.906643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.906607 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-certs\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.907046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.907031 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-certs\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.907874 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.907738 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-data\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.908116 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.908096 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af49f007-5f21-43cc-b4f3-d78e31b45100-istiod-ca-cert\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.908799 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.908778 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-credential-socket\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.909058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.909036 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-socket\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.912177 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.911368 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-podinfo\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.919100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.919077 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-envoy\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.920438 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.920267 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-token\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.921595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.921570 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r6db\" (UniqueName: \"kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-kube-api-access-8r6db\") pod \"e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:46.947102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.947057 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" event={"ID":"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903","Type":"ContainerStarted","Data":"f3eb1ef19d45548ba59c05974de3f515521a241a1d6c23b0c274cea205ac46df"} Oct 30 23:49:46.949284 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.949255 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" event={"ID":"a74a48f3-2b68-424b-9489-33e22dab2f2f","Type":"ContainerStarted","Data":"851e4c679073d2c695b1c06fe3c2af3ff196bc23efffd3ae3aa7db70b8c17ad8"} Oct 30 23:49:46.949548 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.949306 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" event={"ID":"a74a48f3-2b68-424b-9489-33e22dab2f2f","Type":"ContainerStarted","Data":"a043f90e7035a4e386bc15823e6385c6a5447d6ef823be9492939d680a4f67cc"} Oct 30 23:49:46.949736 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.949717 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:49:46.951816 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.951776 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" event={"ID":"fb14f0ca-115b-4fe6-afcd-6338df872573","Type":"ContainerStarted","Data":"734e0129e0d5009c2d894c51431c9a5313ff3f54071b2dc5309f546eade589d3"} Oct 30 23:49:46.951945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.951820 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" event={"ID":"fb14f0ca-115b-4fe6-afcd-6338df872573","Type":"ContainerStarted","Data":"675e05d71c07535fcc78e3e3b109c12e74b9a8ac4072bf725391dbe1e0111ac9"} Oct 30 23:49:46.951945 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.951909 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:49:46.974010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:46.973934 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" podStartSLOduration=1.9739154540000001 podStartE2EDuration="1.973915454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:46.96801762 +0000 UTC m=+1798.487018710" watchObservedRunningTime="2026-10-30 23:49:46.973915454 +0000 UTC m=+1798.492916544" Oct 30 23:49:47.002327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:47.002263 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" podStartSLOduration=2.002243863 podStartE2EDuration="2.002243863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:47.000549086 +0000 UTC m=+1798.519550178" watchObservedRunningTime="2026-10-30 23:49:47.002243863 +0000 UTC m=+1798.521244957" Oct 30 23:49:47.166643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:47.166294 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:47.355671 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:47.354266 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s"] Oct 30 23:49:47.355671 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:47.354571 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" podUID="4688133b-ced0-47cc-aac2-e0cf0237ee52" containerName="maas-api" containerID="cri-o://8580508d5ca11d86b887d1b6d100d6b41c8ee355cdc848472a3202440831f137" gracePeriod=30 Oct 30 23:49:47.564290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:47.564040 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk"] Oct 30 23:49:47.580549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:47.578682 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:49:47.581965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:47.580794 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:49:47.581965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:47.580887 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:49:48.024254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.022938 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" event={"ID":"af49f007-5f21-43cc-b4f3-d78e31b45100","Type":"ContainerStarted","Data":"6e10c215485e3d6ccb26655f26754e13b3bcda5a3f83836c46072df0e29cc25a"} Oct 30 23:49:48.024254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.023004 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" event={"ID":"af49f007-5f21-43cc-b4f3-d78e31b45100","Type":"ContainerStarted","Data":"cfbb610c13b3992de772f7c9b258ac48d3e90f25023b56ae157ae0c0d3dd5229"} Oct 30 23:49:48.035701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.033806 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" event={"ID":"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903","Type":"ContainerStarted","Data":"d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9"} Oct 30 23:49:48.035701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.034756 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:48.038114 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.037850 7570 generic.go:356] "Generic (PLEG): container finished" podID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerID="c6a4e1f7e0b4586b2ac7fe2223526016c1728d8d667ebecad7ed2c062d5eae65" exitCode=0 Oct 30 23:49:48.038114 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.038054 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" event={"ID":"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f","Type":"ContainerDied","Data":"c6a4e1f7e0b4586b2ac7fe2223526016c1728d8d667ebecad7ed2c062d5eae65"} Oct 30 23:49:48.042992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.041285 7570 generic.go:356] "Generic (PLEG): container finished" podID="4688133b-ced0-47cc-aac2-e0cf0237ee52" containerID="8580508d5ca11d86b887d1b6d100d6b41c8ee355cdc848472a3202440831f137" exitCode=0 Oct 30 23:49:48.042992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.042288 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" event={"ID":"4688133b-ced0-47cc-aac2-e0cf0237ee52","Type":"ContainerDied","Data":"8580508d5ca11d86b887d1b6d100d6b41c8ee355cdc848472a3202440831f137"} Oct 30 23:49:48.043387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.043371 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:49:48.081705 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.081652 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" podStartSLOduration=2.081637277 podStartE2EDuration="2.081637277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:48.071931402 +0000 UTC m=+1799.590932495" watchObservedRunningTime="2026-10-30 23:49:48.081637277 +0000 UTC m=+1799.600638440" Oct 30 23:49:48.095667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.094776 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podStartSLOduration=3.522681531 podStartE2EDuration="4.094756761s" totalImagesPullingTime="572.07523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:48.09434006 +0000 UTC m=+1799.613341150" watchObservedRunningTime="2026-10-30 23:49:48.094756761 +0000 UTC m=+1799.613757851" Oct 30 23:49:48.137108 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.132877 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:48.173046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.172562 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:48.173046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.172679 7570 patch_prober.go:29] interesting pod/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 23:49:48.173046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.172725 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.185264 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-maas-api-tls\") pod \"4688133b-ced0-47cc-aac2-e0cf0237ee52\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.185343 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-socket\") pod \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.185562 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-envoy\") pod \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.185615 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-kube-api-access-lgpbv\" (UniqueName: \"kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-kube-api-access-lgpbv\") pod \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.185661 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4688133b-ced0-47cc-aac2-e0cf0237ee52-kube-api-access-hmwcv\" (UniqueName: \"kubernetes.io/projected/4688133b-ced0-47cc-aac2-e0cf0237ee52-kube-api-access-hmwcv\") pod \"4688133b-ced0-47cc-aac2-e0cf0237ee52\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.185694 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-metrics-certs\") pod \"4688133b-ced0-47cc-aac2-e0cf0237ee52\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.185725 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-token\" (UniqueName: \"kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-token\") pod \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.185766 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-certs\") pod \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.185922 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istiod-ca-cert\") pod \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.186089 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4688133b-ced0-47cc-aac2-e0cf0237ee52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688133b-ced0-47cc-aac2-e0cf0237ee52-openshift-service-ca\") pod \"4688133b-ced0-47cc-aac2-e0cf0237ee52\" (UID: \"4688133b-ced0-47cc-aac2-e0cf0237ee52\") " Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.186158 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-data\") pod \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.186191 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-credential-socket\") pod \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " Oct 30 23:49:48.186863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.186231 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-podinfo\") pod \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\" (UID: \"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f\") " Oct 30 23:49:48.189234 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.189213 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istiod-ca-cert" pod "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" (UID: "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:49:48.189993 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.189972 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-socket" pod "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" (UID: "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:48.192348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.191110 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4688133b-ced0-47cc-aac2-e0cf0237ee52-openshift-service-ca" pod "4688133b-ced0-47cc-aac2-e0cf0237ee52" (UID: "4688133b-ced0-47cc-aac2-e0cf0237ee52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:49:48.192348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.191653 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-data" pod "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" (UID: "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:48.192348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.191957 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-credential-socket" pod "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" (UID: "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:48.192717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.192656 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-certs" pod "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" (UID: "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:48.194742 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.194705 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-podinfo" pod "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" (UID: "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:49:48.195081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.195053 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-maas-api-tls" pod "4688133b-ced0-47cc-aac2-e0cf0237ee52" (UID: "4688133b-ced0-47cc-aac2-e0cf0237ee52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:49:48.195554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.195481 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-envoy" pod "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" (UID: "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:48.195554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.195520 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-token" pod "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" (UID: "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:48.196052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.196000 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-kube-api-access-lgpbv" pod "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" (UID: "0968c2d5-5fa4-418c-b11b-0f1cf8cde86f"). InnerVolumeSpecName "kube-api-access-lgpbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:48.196214 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.196175 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-metrics-certs" pod "4688133b-ced0-47cc-aac2-e0cf0237ee52" (UID: "4688133b-ced0-47cc-aac2-e0cf0237ee52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:49:48.196431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.196364 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4688133b-ced0-47cc-aac2-e0cf0237ee52-kube-api-access-hmwcv" pod "4688133b-ced0-47cc-aac2-e0cf0237ee52" (UID: "4688133b-ced0-47cc-aac2-e0cf0237ee52"). InnerVolumeSpecName "kube-api-access-hmwcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300534 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300576 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688133b-ced0-47cc-aac2-e0cf0237ee52-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300590 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300603 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300621 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300633 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300649 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300661 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300675 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgpbv\" (UniqueName: \"kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-kube-api-access-lgpbv\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300690 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmwcv\" (UniqueName: \"kubernetes.io/projected/4688133b-ced0-47cc-aac2-e0cf0237ee52-kube-api-access-hmwcv\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300705 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4688133b-ced0-47cc-aac2-e0cf0237ee52-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300717 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.302494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.300730 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:49:48.839212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.839181 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:49:48.839595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:48.839572 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:49:49.055052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:49.054975 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" Oct 30 23:49:49.055052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:49.054996 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2" event={"ID":"0968c2d5-5fa4-418c-b11b-0f1cf8cde86f","Type":"ContainerDied","Data":"8d8209ac4e8f282dba722f237b62f035589e245b00c153fb5807e8ab1ec20e67"} Oct 30 23:49:49.055755 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:49.055089 7570 scope.go:122] "RemoveContainer" containerID="c6a4e1f7e0b4586b2ac7fe2223526016c1728d8d667ebecad7ed2c062d5eae65" Oct 30 23:49:49.058148 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:49.056992 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" event={"ID":"4688133b-ced0-47cc-aac2-e0cf0237ee52","Type":"ContainerDied","Data":"8ba4ff04e71c48c24fa6655fb5f404187cad3ec935c377a8a6113dab43851839"} Oct 30 23:49:49.058148 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:49.057047 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s" Oct 30 23:49:49.102487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:49.101612 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s"] Oct 30 23:49:49.113775 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:49.106915 7570 scope.go:122] "RemoveContainer" containerID="8580508d5ca11d86b887d1b6d100d6b41c8ee355cdc848472a3202440831f137" Oct 30 23:49:49.113775 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:49.109688 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a162557-55c4486749-qp69s"] Oct 30 23:49:49.129624 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:49.129590 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2"] Oct 30 23:49:49.139393 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:49.139361 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-nxpl2"] Oct 30 23:49:49.176540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:49.176512 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:50.070111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.070080 7570 generic.go:356] "Generic (PLEG): container finished" podID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerID="982b2a7324033b2bc083f03b651ac5a85ec2d52e6b10437d961438545835dea8" exitCode=0 Oct 30 23:49:50.070615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.070156 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" event={"ID":"51164b20-de0a-4b55-bdb7-7c25c2d5d922","Type":"ContainerDied","Data":"982b2a7324033b2bc083f03b651ac5a85ec2d52e6b10437d961438545835dea8"} Oct 30 23:49:50.070697 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.070676 7570 scope.go:122] "RemoveContainer" containerID="982b2a7324033b2bc083f03b651ac5a85ec2d52e6b10437d961438545835dea8" Oct 30 23:49:50.071793 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.071701 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:50.072687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.072671 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:49:50.072831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.072805 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:49:50.072885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.072871 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:49:50.072925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.072905 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:49:50.607000 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.606969 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" path="/var/lib/kubelet/pods/0968c2d5-5fa4-418c-b11b-0f1cf8cde86f/volumes" Oct 30 23:49:50.607625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.607608 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4688133b-ced0-47cc-aac2-e0cf0237ee52" path="/var/lib/kubelet/pods/4688133b-ced0-47cc-aac2-e0cf0237ee52/volumes" Oct 30 23:49:50.906130 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.906033 7570 scope.go:122] "RemoveContainer" containerID="f4e91fc31d2b376736a04c73e73a1f18650434fe6d917af29a8dfe4e51814e52" Oct 30 23:49:50.918150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.918128 7570 scope.go:122] "RemoveContainer" containerID="4de508c96450866d11d19194631173a6e4839f71f9ba4cb506c7376a37938698" Oct 30 23:49:50.932845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.932821 7570 scope.go:122] "RemoveContainer" containerID="6e6e36bf126f356a280d07b7eec337ee6e6a4f1456d14fd41ac96c86f261d398" Oct 30 23:49:50.964076 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:50.964043 7570 scope.go:122] "RemoveContainer" containerID="486be85973c7f9a10c7b572e17dacd1749f4123d0fe7af825eb3a404120fc545" Oct 30 23:49:51.084182 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.084152 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" event={"ID":"51164b20-de0a-4b55-bdb7-7c25c2d5d922","Type":"ContainerStarted","Data":"94c4b43eba16793a73bbb2eec7436a8d641d6293bbd05ef1dc2359a826b91d3d"} Oct 30 23:49:51.159300 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.159248 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:49:51.160739 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.160716 7570 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 23:49:51.160841 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.160761 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 23:49:51.871628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.871588 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght"] Oct 30 23:49:51.872361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.872348 7570 cpu_manager.go:414] "containerMap: removing container" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" Oct 30 23:49:51.872423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.872363 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" Oct 30 23:49:51.872478 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.872437 7570 cpu_manager.go:414] "containerMap: removing container" podUID="4688133b-ced0-47cc-aac2-e0cf0237ee52" containerName="maas-api" Oct 30 23:49:51.872478 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.872444 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4688133b-ced0-47cc-aac2-e0cf0237ee52" containerName="maas-api" Oct 30 23:49:51.872582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.872565 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="0968c2d5-5fa4-418c-b11b-0f1cf8cde86f" containerName="istio-proxy" Oct 30 23:49:51.872634 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.872588 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4688133b-ced0-47cc-aac2-e0cf0237ee52" containerName="maas-api" Oct 30 23:49:51.877875 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.877853 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:49:51.909495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.897496 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-82d449\"" Oct 30 23:49:51.909495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.897715 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-82d449-dockercfg-fwwd9\"" Oct 30 23:49:51.925151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.925121 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght"] Oct 30 23:49:51.941691 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.937849 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g"] Oct 30 23:49:51.958360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.948928 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:49:51.986493 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:51.966705 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g"] Oct 30 23:49:52.117556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.100244 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv58n\" (UniqueName: \"kubernetes.io/projected/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-kube-api-access-pv58n\") pod \"payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g\" (UID: \"aa08b804-5246-4fb8-a68c-cf7af9ba4a54\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:49:52.164745 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.161305 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpr7l\" (UniqueName: \"kubernetes.io/projected/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-kube-api-access-vpr7l\") pod \"payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght\" (UID: \"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:49:52.164745 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.161377 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght\" (UID: \"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:49:52.164745 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.161645 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g\" (UID: \"aa08b804-5246-4fb8-a68c-cf7af9ba4a54\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:49:52.182107 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.166773 7570 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 23:49:52.182107 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.166848 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 23:49:52.272842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.272799 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv58n\" (UniqueName: \"kubernetes.io/projected/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-kube-api-access-pv58n\") pod \"payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g\" (UID: \"aa08b804-5246-4fb8-a68c-cf7af9ba4a54\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:49:52.274238 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.273193 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpr7l\" (UniqueName: \"kubernetes.io/projected/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-kube-api-access-vpr7l\") pod \"payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght\" (UID: \"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:49:52.274238 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.273249 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght\" (UID: \"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:49:52.274238 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.273394 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g\" (UID: \"aa08b804-5246-4fb8-a68c-cf7af9ba4a54\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:49:52.274543 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.274314 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g\" (UID: \"aa08b804-5246-4fb8-a68c-cf7af9ba4a54\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:49:52.275414 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.275389 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght\" (UID: \"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:49:52.318215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.318183 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpr7l\" (UniqueName: \"kubernetes.io/projected/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-kube-api-access-vpr7l\") pod \"payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght\" (UID: \"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:49:52.318858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.318836 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv58n\" (UniqueName: \"kubernetes.io/projected/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-kube-api-access-pv58n\") pod \"payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g\" (UID: \"aa08b804-5246-4fb8-a68c-cf7af9ba4a54\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:49:52.536792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.536674 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:49:52.573526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.572109 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:49:52.932545 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.932518 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght"] Oct 30 23:49:52.933904 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:49:52.933871 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e6db40_bf7c_4440_bed0_8ec6e70cab0f.slice/crio-4b86cd137d561736b49011f33a2776b4b31a08a41a2dd50f166c30ae6407bbc8 WatchSource:0}: Error finding container 4b86cd137d561736b49011f33a2776b4b31a08a41a2dd50f166c30ae6407bbc8: Status 404 returned error can't find the container with id 4b86cd137d561736b49011f33a2776b4b31a08a41a2dd50f166c30ae6407bbc8 Oct 30 23:49:52.957883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:52.957857 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g"] Oct 30 23:49:52.958659 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:49:52.958637 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa08b804_5246_4fb8_a68c_cf7af9ba4a54.slice/crio-292e48f0a27d0eb592da0c973a7bf973bbc8b4305edc93f0165f8f89486979b2 WatchSource:0}: Error finding container 292e48f0a27d0eb592da0c973a7bf973bbc8b4305edc93f0165f8f89486979b2: Status 404 returned error can't find the container with id 292e48f0a27d0eb592da0c973a7bf973bbc8b4305edc93f0165f8f89486979b2 Oct 30 23:49:53.159586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:53.159550 7570 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 23:49:53.159998 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:53.159608 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 23:49:53.305094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:53.304996 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" event={"ID":"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f","Type":"ContainerStarted","Data":"0170737f67453b691218a3069bf111fefb4d960cddcbbb0718f56f6f8b6dd3b2"} Oct 30 23:49:53.305342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:53.305318 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" event={"ID":"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f","Type":"ContainerStarted","Data":"4b86cd137d561736b49011f33a2776b4b31a08a41a2dd50f166c30ae6407bbc8"} Oct 30 23:49:53.311146 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:53.311084 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:49:53.313827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:53.313798 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" event={"ID":"aa08b804-5246-4fb8-a68c-cf7af9ba4a54","Type":"ContainerStarted","Data":"4c9d727f8c05504972593ae1b6240121490aba70db54418fa228de5c0ea97248"} Oct 30 23:49:53.313964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:53.313849 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" event={"ID":"aa08b804-5246-4fb8-a68c-cf7af9ba4a54","Type":"ContainerStarted","Data":"292e48f0a27d0eb592da0c973a7bf973bbc8b4305edc93f0165f8f89486979b2"} Oct 30 23:49:53.313964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:53.313950 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:49:53.336075 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:53.336021 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" podStartSLOduration=2.336002702 podStartE2EDuration="2.336002702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:53.329787382 +0000 UTC m=+1804.848788473" watchObservedRunningTime="2026-10-30 23:49:53.336002702 +0000 UTC m=+1804.855003794" Oct 30 23:49:53.355357 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:53.355298 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" podStartSLOduration=2.355284985 podStartE2EDuration="2.355284985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:53.352397097 +0000 UTC m=+1804.871398190" watchObservedRunningTime="2026-10-30 23:49:53.355284985 +0000 UTC m=+1804.874286089" Oct 30 23:49:53.627143 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:53.627107 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:49:54.159471 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:54.159423 7570 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 23:49:54.159648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:54.159505 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 23:49:55.078864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:55.078829 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:49:55.159607 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:55.159566 7570 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 23:49:55.159813 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:55.159632 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 23:49:55.856233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:55.856201 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc"] Oct 30 23:49:55.861320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:55.861214 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:55.872517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:55.870503 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a63288-serving-cert\"" Oct 30 23:49:55.884534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:55.884454 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc"] Oct 30 23:49:55.982691 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:55.982651 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:55.982923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:55.982789 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-metrics-certs\") pod \"maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:55.982923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:55.982845 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-maas-api-tls\") pod \"maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:55.983090 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:55.983073 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcbvg\" (UniqueName: \"kubernetes.io/projected/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-kube-api-access-xcbvg\") pod \"maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:56.095150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:56.084616 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-metrics-certs\") pod \"maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:56.095150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:56.084717 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-maas-api-tls\") pod \"maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:56.095150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:56.085017 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcbvg\" (UniqueName: \"kubernetes.io/projected/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-kube-api-access-xcbvg\") pod \"maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:56.095150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:56.085093 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:56.107535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:56.100577 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:56.109496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:56.108927 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-maas-api-tls\") pod \"maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:56.142932 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:56.142218 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-metrics-certs\") pod \"maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:56.142932 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:56.142854 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcbvg\" (UniqueName: \"kubernetes.io/projected/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-kube-api-access-xcbvg\") pod \"maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:56.159756 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:56.159725 7570 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 23:49:56.159958 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:56.159936 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 23:49:56.195487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:56.191670 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:56.524541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:56.524515 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc"] Oct 30 23:49:56.526721 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:49:56.526688 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3d259c_4f6c_4033_b6dd_1059616fe3b4.slice/crio-e2166af4fede7416e20d757b1ece44a9283660c9ee1544125db3c86378b99b4f WatchSource:0}: Error finding container e2166af4fede7416e20d757b1ece44a9283660c9ee1544125db3c86378b99b4f: Status 404 returned error can't find the container with id e2166af4fede7416e20d757b1ece44a9283660c9ee1544125db3c86378b99b4f Oct 30 23:49:57.164419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:57.164392 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:49:57.165666 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:57.165646 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:49:57.357552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:57.357503 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" event={"ID":"6b3d259c-4f6c-4033-b6dd-1059616fe3b4","Type":"ContainerStarted","Data":"a122980b7cbc76a173c9f98de07c0e43bb457617d98005f8f366d95243978ad7"} Oct 30 23:49:57.357756 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:57.357569 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" event={"ID":"6b3d259c-4f6c-4033-b6dd-1059616fe3b4","Type":"ContainerStarted","Data":"e2166af4fede7416e20d757b1ece44a9283660c9ee1544125db3c86378b99b4f"} Oct 30 23:49:57.357840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:57.357760 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:49:57.381341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:57.381291 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podStartSLOduration=1.941944095 podStartE2EDuration="2.381277535s" totalImagesPullingTime="439.33344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:57.377422906 +0000 UTC m=+1808.896423997" watchObservedRunningTime="2026-10-30 23:49:57.381277535 +0000 UTC m=+1808.900278624" Oct 30 23:49:58.045536 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:58.045494 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:49:58.045946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:49:58.045927 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:49:59.426422 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:49:59.425181 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:50:03.373075 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:03.373045 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:50:04.330167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:04.330137 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:50:05.338172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:05.338146 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:50:15.803976 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:15.803938 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl"] Oct 30 23:50:15.808944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:15.808927 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:50:15.818901 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:15.818867 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f0817c\"" Oct 30 23:50:15.822511 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:15.822481 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl"] Oct 30 23:50:15.914065 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:15.914012 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7chn\" (UniqueName: \"kubernetes.io/projected/a4acfbb3-3ee2-4890-984a-2fc8a796bece-kube-api-access-b7chn\") pod \"payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl\" (UID: \"a4acfbb3-3ee2-4890-984a-2fc8a796bece\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:50:15.914251 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:15.914130 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4acfbb3-3ee2-4890-984a-2fc8a796bece-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl\" (UID: \"a4acfbb3-3ee2-4890-984a-2fc8a796bece\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:50:16.046648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:16.046607 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7chn\" (UniqueName: \"kubernetes.io/projected/a4acfbb3-3ee2-4890-984a-2fc8a796bece-kube-api-access-b7chn\") pod \"payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl\" (UID: \"a4acfbb3-3ee2-4890-984a-2fc8a796bece\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:50:16.050662 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:16.050619 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4acfbb3-3ee2-4890-984a-2fc8a796bece-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl\" (UID: \"a4acfbb3-3ee2-4890-984a-2fc8a796bece\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:50:16.079912 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:16.079543 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4acfbb3-3ee2-4890-984a-2fc8a796bece-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl\" (UID: \"a4acfbb3-3ee2-4890-984a-2fc8a796bece\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:50:16.125489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:16.119725 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7chn\" (UniqueName: \"kubernetes.io/projected/a4acfbb3-3ee2-4890-984a-2fc8a796bece-kube-api-access-b7chn\") pod \"payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl\" (UID: \"a4acfbb3-3ee2-4890-984a-2fc8a796bece\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:50:16.137143 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:16.137105 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:50:16.484804 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:50:16.484771 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4acfbb3_3ee2_4890_984a_2fc8a796bece.slice/crio-4fe5703ab82627b5845517038cf50e6b42c7ac86c43b503bddefc6a123c89810 WatchSource:0}: Error finding container 4fe5703ab82627b5845517038cf50e6b42c7ac86c43b503bddefc6a123c89810: Status 404 returned error can't find the container with id 4fe5703ab82627b5845517038cf50e6b42c7ac86c43b503bddefc6a123c89810 Oct 30 23:50:16.485194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:16.485168 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl"] Oct 30 23:50:16.549106 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:16.549078 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" event={"ID":"a4acfbb3-3ee2-4890-984a-2fc8a796bece","Type":"ContainerStarted","Data":"4fe5703ab82627b5845517038cf50e6b42c7ac86c43b503bddefc6a123c89810"} Oct 30 23:50:17.559909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:17.559873 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" event={"ID":"a4acfbb3-3ee2-4890-984a-2fc8a796bece","Type":"ContainerStarted","Data":"3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821"} Oct 30 23:50:17.560423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:17.559958 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:50:17.580561 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:17.580497 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" podStartSLOduration=2.580484343 podStartE2EDuration="2.580484343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:17.579060215 +0000 UTC m=+1829.098061305" watchObservedRunningTime="2026-10-30 23:50:17.580484343 +0000 UTC m=+1829.099485433" Oct 30 23:50:21.031592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.031556 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69"] Oct 30 23:50:21.036941 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.036916 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.055614 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.055580 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69"] Oct 30 23:50:21.089887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.089852 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-token\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.090038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.089951 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-data\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.090038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.090010 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istiod-ca-cert\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.090149 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.090037 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnsm4\" (UniqueName: \"kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-kube-api-access-qnsm4\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.090149 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.090087 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-socket\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.090149 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.090114 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-certs\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.090265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.090220 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-envoy\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.090567 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.090548 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-credential-socket\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.090648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.090622 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-podinfo\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.112834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.112799 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-32732d-openshift-default-dockercfg-fvnxc\"" Oct 30 23:50:21.299587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.286819 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-credential-socket\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.299587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.286915 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-podinfo\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.299587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.286974 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-token\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.299587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.287011 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-data\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.299587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.287059 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istiod-ca-cert\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.299587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.287081 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnsm4\" (UniqueName: \"kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-kube-api-access-qnsm4\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.299587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.287130 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-socket\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.299587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.287160 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-certs\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.299587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.287265 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-envoy\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.309486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.304773 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-data\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.309486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.305039 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-credential-socket\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.313487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.310683 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istiod-ca-cert\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.339488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.339397 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnsm4\" (UniqueName: \"kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-kube-api-access-qnsm4\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.354488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.354286 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-socket\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.412627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.412564 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-certs\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.425129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.421771 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-envoy\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.425129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.422711 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-podinfo\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.438053 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.438020 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-token\") pod \"e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:21.676802 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:21.676768 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:22.034988 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:22.034958 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69"] Oct 30 23:50:22.039037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:22.038993 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:50:22.039250 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:22.039228 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:50:22.039398 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:22.039379 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:50:22.606642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:22.606595 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" event={"ID":"84a7842f-6043-4b93-ab32-d25e2ba8c5bb","Type":"ContainerStarted","Data":"de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb"} Oct 30 23:50:22.606642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:22.606641 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" event={"ID":"84a7842f-6043-4b93-ab32-d25e2ba8c5bb","Type":"ContainerStarted","Data":"10e5e2a6307b59b62b3433b9f7fcd74596640f4cfa642f931e0d87d3f4749f90"} Oct 30 23:50:22.639009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:22.638947 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" podStartSLOduration=2.638930829 podStartE2EDuration="2.638930829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:22.633534386 +0000 UTC m=+1834.152535477" watchObservedRunningTime="2026-10-30 23:50:22.638930829 +0000 UTC m=+1834.157931920" Oct 30 23:50:22.677394 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:22.677366 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:23.683757 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:23.683730 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:24.625158 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:24.625131 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:24.626178 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:24.626158 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:50:27.654488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:27.654432 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg"] Oct 30 23:50:27.664025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:27.663633 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:50:27.670997 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:27.670972 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg"] Oct 30 23:50:27.689517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:27.688995 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-32732d-dockercfg-rcltc\"" Oct 30 23:50:27.703720 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:27.703678 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg\" (UID: \"37d7f8cf-f311-4b98-a6b7-25a043ac78ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:50:27.704117 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:27.704087 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tchs\" (UniqueName: \"kubernetes.io/projected/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-kube-api-access-5tchs\") pod \"payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg\" (UID: \"37d7f8cf-f311-4b98-a6b7-25a043ac78ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:50:27.715484 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:27.715444 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-32732d\"" Oct 30 23:50:27.810485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:27.808235 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tchs\" (UniqueName: \"kubernetes.io/projected/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-kube-api-access-5tchs\") pod \"payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg\" (UID: \"37d7f8cf-f311-4b98-a6b7-25a043ac78ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:50:27.810485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:27.808574 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg\" (UID: \"37d7f8cf-f311-4b98-a6b7-25a043ac78ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:50:27.810485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:27.809696 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg\" (UID: \"37d7f8cf-f311-4b98-a6b7-25a043ac78ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:50:27.886486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:27.885594 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tchs\" (UniqueName: \"kubernetes.io/projected/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-kube-api-access-5tchs\") pod \"payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg\" (UID: \"37d7f8cf-f311-4b98-a6b7-25a043ac78ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:50:27.995537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:27.991666 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:50:28.458608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:28.456624 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg"] Oct 30 23:50:28.466869 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:50:28.461676 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d7f8cf_f311_4b98_a6b7_25a043ac78ae.slice/crio-71d638e5a06a653cff071eabe2527b1dada7d93894db7e64b3cd387645c9f920 WatchSource:0}: Error finding container 71d638e5a06a653cff071eabe2527b1dada7d93894db7e64b3cd387645c9f920: Status 404 returned error can't find the container with id 71d638e5a06a653cff071eabe2527b1dada7d93894db7e64b3cd387645c9f920 Oct 30 23:50:28.573059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:28.573015 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:50:28.683224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:28.683194 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" event={"ID":"37d7f8cf-f311-4b98-a6b7-25a043ac78ae","Type":"ContainerStarted","Data":"71d638e5a06a653cff071eabe2527b1dada7d93894db7e64b3cd387645c9f920"} Oct 30 23:50:29.565049 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:50:29.565006 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:50:29.758582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:29.745849 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" event={"ID":"37d7f8cf-f311-4b98-a6b7-25a043ac78ae","Type":"ContainerStarted","Data":"a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7"} Oct 30 23:50:29.758582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:29.758573 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:50:40.054635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.054571 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" podStartSLOduration=13.054553605 podStartE2EDuration="13.054553605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:29.777727899 +0000 UTC m=+1841.296729002" watchObservedRunningTime="2026-10-30 23:50:40.054553605 +0000 UTC m=+1851.573554696" Oct 30 23:50:40.060438 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.060401 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24"] Oct 30 23:50:40.156588 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.156560 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24"] Oct 30 23:50:40.156766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.156655 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.326010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.325921 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.326178 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.326081 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-envoy\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.326178 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.326148 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-podinfo\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.326284 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.326181 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m875z\" (UniqueName: \"kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-kube-api-access-m875z\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.326284 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.326276 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-certs\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.326399 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.326387 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94094c6-876a-4082-ad7d-f35bb1909f87-istiod-ca-cert\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.326446 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.326407 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-data\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.326560 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.326542 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-credential-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.326606 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.326584 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-token\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.428236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.428200 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94094c6-876a-4082-ad7d-f35bb1909f87-istiod-ca-cert\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.428236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.428249 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-data\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.428590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.428423 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-credential-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.428590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.428490 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-token\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.428590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.428549 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.428776 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.428664 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-envoy\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.428776 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.428690 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-podinfo\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.428895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.428775 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m875z\" (UniqueName: \"kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-kube-api-access-m875z\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.428895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.428794 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-data\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.428895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.428843 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-credential-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.428895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.428861 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-certs\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.429113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.429012 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.429113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.429062 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94094c6-876a-4082-ad7d-f35bb1909f87-istiod-ca-cert\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.429638 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.429591 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-certs\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.432183 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.432152 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-podinfo\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.432844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.432808 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-envoy\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.443513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.443444 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-token\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.443513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.443447 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m875z\" (UniqueName: \"kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-kube-api-access-m875z\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.539538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.539506 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:40.762115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.761535 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:50:40.918085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.918055 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24"] Oct 30 23:50:40.918713 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:50:40.918688 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94094c6_876a_4082_ad7d_f35bb1909f87.slice/crio-5307084d1930531fb5103a70df19b2a4a36153c9a10084188240540440dd538c WatchSource:0}: Error finding container 5307084d1930531fb5103a70df19b2a4a36153c9a10084188240540440dd538c: Status 404 returned error can't find the container with id 5307084d1930531fb5103a70df19b2a4a36153c9a10084188240540440dd538c Oct 30 23:50:40.921947 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.921912 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:50:40.922055 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.921986 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:50:40.922055 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:40.922021 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:50:41.869818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:41.869779 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" event={"ID":"b94094c6-876a-4082-ad7d-f35bb1909f87","Type":"ContainerStarted","Data":"afc2d23e0695cedbf256b4a7750b95ea9fcba96be1a1b86cbbbaa52d2c4a5cf5"} Oct 30 23:50:41.870315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:41.869880 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" event={"ID":"b94094c6-876a-4082-ad7d-f35bb1909f87","Type":"ContainerStarted","Data":"5307084d1930531fb5103a70df19b2a4a36153c9a10084188240540440dd538c"} Oct 30 23:50:41.898498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:41.898433 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podStartSLOduration=1.8984212390000001 podStartE2EDuration="1.898421239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:41.894717282 +0000 UTC m=+1853.413718372" watchObservedRunningTime="2026-10-30 23:50:41.898421239 +0000 UTC m=+1853.417422329" Oct 30 23:50:42.271581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.269107 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc"] Oct 30 23:50:42.348411 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.348373 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws"] Oct 30 23:50:42.350581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.350554 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:50:42.360951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.360811 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7b0299\"" Oct 30 23:50:42.364379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.363615 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc"] Oct 30 23:50:42.364379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.363661 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws"] Oct 30 23:50:42.364379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.363744 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:50:42.427974 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.427310 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6df9z\" (UniqueName: \"kubernetes.io/projected/5244915c-9573-489c-b1f8-e22948899df3-kube-api-access-6df9z\") pod \"payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws\" (UID: \"5244915c-9573-489c-b1f8-e22948899df3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:50:42.427974 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.427557 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5244915c-9573-489c-b1f8-e22948899df3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws\" (UID: \"5244915c-9573-489c-b1f8-e22948899df3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:50:42.427974 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.427678 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12efb180-e881-40a3-9001-8a39a2ddd352-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc\" (UID: \"12efb180-e881-40a3-9001-8a39a2ddd352\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:50:42.427974 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.427786 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjznr\" (UniqueName: \"kubernetes.io/projected/12efb180-e881-40a3-9001-8a39a2ddd352-kube-api-access-cjznr\") pod \"payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc\" (UID: \"12efb180-e881-40a3-9001-8a39a2ddd352\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:50:42.586376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.554063 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5244915c-9573-489c-b1f8-e22948899df3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws\" (UID: \"5244915c-9573-489c-b1f8-e22948899df3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:50:42.586376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.554197 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12efb180-e881-40a3-9001-8a39a2ddd352-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc\" (UID: \"12efb180-e881-40a3-9001-8a39a2ddd352\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:50:42.586376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.554295 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjznr\" (UniqueName: \"kubernetes.io/projected/12efb180-e881-40a3-9001-8a39a2ddd352-kube-api-access-cjznr\") pod \"payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc\" (UID: \"12efb180-e881-40a3-9001-8a39a2ddd352\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:50:42.586376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.554415 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6df9z\" (UniqueName: \"kubernetes.io/projected/5244915c-9573-489c-b1f8-e22948899df3-kube-api-access-6df9z\") pod \"payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws\" (UID: \"5244915c-9573-489c-b1f8-e22948899df3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:50:42.586376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.555840 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5244915c-9573-489c-b1f8-e22948899df3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws\" (UID: \"5244915c-9573-489c-b1f8-e22948899df3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:50:42.586376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.556572 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12efb180-e881-40a3-9001-8a39a2ddd352-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc\" (UID: \"12efb180-e881-40a3-9001-8a39a2ddd352\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:50:42.627489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.617877 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:42.641520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.641278 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 23:50:42.641520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.641335 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 23:50:42.647182 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.643679 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6df9z\" (UniqueName: \"kubernetes.io/projected/5244915c-9573-489c-b1f8-e22948899df3-kube-api-access-6df9z\") pod \"payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws\" (UID: \"5244915c-9573-489c-b1f8-e22948899df3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:50:42.647182 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.645428 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjznr\" (UniqueName: \"kubernetes.io/projected/12efb180-e881-40a3-9001-8a39a2ddd352-kube-api-access-cjznr\") pod \"payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc\" (UID: \"12efb180-e881-40a3-9001-8a39a2ddd352\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:50:42.769509 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.769471 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:50:42.828485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:42.810320 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:50:43.324974 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.321746 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws"] Oct 30 23:50:43.324974 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:50:43.324726 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12efb180_e881_40a3_9001_8a39a2ddd352.slice/crio-e31e4adbff9fdede5fa5e5267e5d0855347ca5ab40d7269790fdb9af9b1da01d WatchSource:0}: Error finding container e31e4adbff9fdede5fa5e5267e5d0855347ca5ab40d7269790fdb9af9b1da01d: Status 404 returned error can't find the container with id e31e4adbff9fdede5fa5e5267e5d0855347ca5ab40d7269790fdb9af9b1da01d Oct 30 23:50:43.325939 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.325899 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc"] Oct 30 23:50:43.546631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.541774 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 23:50:43.546631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.541837 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 23:50:43.919654 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.919609 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:50:43.919832 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.919676 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:50:43.929347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.929316 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" event={"ID":"5244915c-9573-489c-b1f8-e22948899df3","Type":"ContainerStarted","Data":"b3c07a3b101c4549f4210027e795ad239cf58b95d2ed5169b8e57030fb403144"} Oct 30 23:50:43.929528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.929373 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" event={"ID":"5244915c-9573-489c-b1f8-e22948899df3","Type":"ContainerStarted","Data":"8a970351fdd6076d4c0a29a1c6188fc87d517621dee89321299f23405c519104"} Oct 30 23:50:43.929528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.929448 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:50:43.931261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.931238 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" event={"ID":"12efb180-e881-40a3-9001-8a39a2ddd352","Type":"ContainerStarted","Data":"3f64b5a78b92f2c34f681907d33dd8081813506ac9e5ee82e9f44f06a2e6b2fd"} Oct 30 23:50:43.931390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.931285 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" event={"ID":"12efb180-e881-40a3-9001-8a39a2ddd352","Type":"ContainerStarted","Data":"e31e4adbff9fdede5fa5e5267e5d0855347ca5ab40d7269790fdb9af9b1da01d"} Oct 30 23:50:43.931390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.931323 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:50:43.948567 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.948525 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" podStartSLOduration=1.948513114 podStartE2EDuration="1.948513114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:43.9470004 +0000 UTC m=+1855.466001491" watchObservedRunningTime="2026-10-30 23:50:43.948513114 +0000 UTC m=+1855.467514204" Oct 30 23:50:43.969383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:43.969337 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" podStartSLOduration=1.969323962 podStartE2EDuration="1.969323962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:43.965599107 +0000 UTC m=+1855.484600199" watchObservedRunningTime="2026-10-30 23:50:43.969323962 +0000 UTC m=+1855.488325051" Oct 30 23:50:44.540372 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:44.540338 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 23:50:44.540803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:44.540388 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 23:50:45.540233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:45.540199 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 23:50:45.540419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:45.540255 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 23:50:46.540584 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:46.540533 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 23:50:46.541003 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:46.540603 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 23:50:46.829701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:46.829669 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs"] Oct 30 23:50:46.830963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:46.830929 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" containerID="cri-o://d8b2cd330f19776fc3ffc3dadda62011a670aae92fc5df4f7240fcfd3c3777f4" gracePeriod=30 Oct 30 23:50:47.539942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:47.539903 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 23:50:47.540153 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:47.539968 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 23:50:47.701163 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:47.701121 7570 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:50:47.701602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:47.701182 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:50:48.546207 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:48.546177 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:48.546706 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:48.546686 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:48.547392 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:48.547375 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:50:48.802649 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:48.802572 7570 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:50:48.803044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:48.802646 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:50:51.160717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:51.160683 7570 scope.go:122] "RemoveContainer" containerID="6c9e0a88b09e666e45a7aec9871522c3ef1d4d52d36d38765e806d3e60a62489" Oct 30 23:50:51.178208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:51.178183 7570 scope.go:122] "RemoveContainer" containerID="174c565459f7356c3e6849f813386e15e436907c1a1ee5ecb6a574c8d2eb2626" Oct 30 23:50:51.190615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:51.190590 7570 scope.go:122] "RemoveContainer" containerID="d2a4ba1529eb00d22b53bbdbb6fcdf460c9fb8070cc9948d8cfd49316f00de13" Oct 30 23:50:51.221894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:51.221871 7570 scope.go:122] "RemoveContainer" containerID="e5e05ed1a9f747e172622be8a80a5fbb22b200bab07b6c10ee18d3173f4243b7" Oct 30 23:50:51.246332 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:51.246308 7570 scope.go:122] "RemoveContainer" containerID="de1c20c3bbac6eaa4680dd5690e282a6612049cdb4837d9b4414afd130212c06" Oct 30 23:50:52.045036 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.045008 7570 generic.go:356] "Generic (PLEG): container finished" podID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerID="d8b2cd330f19776fc3ffc3dadda62011a670aae92fc5df4f7240fcfd3c3777f4" exitCode=0 Oct 30 23:50:52.045209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.045095 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" event={"ID":"b4e91d97-e0c2-4406-b78a-e4701515b350","Type":"ContainerDied","Data":"d8b2cd330f19776fc3ffc3dadda62011a670aae92fc5df4f7240fcfd3c3777f4"} Oct 30 23:50:52.045209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.045178 7570 scope.go:122] "RemoveContainer" containerID="3e989089df3c985b7c299b5fd8d0aa156cc68cef18dfd591aaad45c558bd396f" Oct 30 23:50:52.346673 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.346635 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:50:52.447489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.446273 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-envoy\") pod \"b4e91d97-e0c2-4406-b78a-e4701515b350\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " Oct 30 23:50:52.447489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.446327 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-kube-api-access-p4h2m\" (UniqueName: \"kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-kube-api-access-p4h2m\") pod \"b4e91d97-e0c2-4406-b78a-e4701515b350\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " Oct 30 23:50:52.447489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.446428 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-socket\") pod \"b4e91d97-e0c2-4406-b78a-e4701515b350\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " Oct 30 23:50:52.447489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.446494 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-credential-socket\") pod \"b4e91d97-e0c2-4406-b78a-e4701515b350\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " Oct 30 23:50:52.447489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.446657 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-podinfo\") pod \"b4e91d97-e0c2-4406-b78a-e4701515b350\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " Oct 30 23:50:52.447489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.446739 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-token\" (UniqueName: \"kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-token\") pod \"b4e91d97-e0c2-4406-b78a-e4701515b350\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " Oct 30 23:50:52.447489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.446776 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-certs\") pod \"b4e91d97-e0c2-4406-b78a-e4701515b350\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " Oct 30 23:50:52.447489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.446925 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-data\") pod \"b4e91d97-e0c2-4406-b78a-e4701515b350\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " Oct 30 23:50:52.447489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.447061 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e91d97-e0c2-4406-b78a-e4701515b350-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e91d97-e0c2-4406-b78a-e4701515b350-istiod-ca-cert\") pod \"b4e91d97-e0c2-4406-b78a-e4701515b350\" (UID: \"b4e91d97-e0c2-4406-b78a-e4701515b350\") " Oct 30 23:50:52.448272 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.448209 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e91d97-e0c2-4406-b78a-e4701515b350-istiod-ca-cert" pod "b4e91d97-e0c2-4406-b78a-e4701515b350" (UID: "b4e91d97-e0c2-4406-b78a-e4701515b350"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:50:52.451044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.450760 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-certs" pod "b4e91d97-e0c2-4406-b78a-e4701515b350" (UID: "b4e91d97-e0c2-4406-b78a-e4701515b350"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:50:52.451757 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.451729 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-data" pod "b4e91d97-e0c2-4406-b78a-e4701515b350" (UID: "b4e91d97-e0c2-4406-b78a-e4701515b350"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:50:52.452268 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.452248 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-socket" pod "b4e91d97-e0c2-4406-b78a-e4701515b350" (UID: "b4e91d97-e0c2-4406-b78a-e4701515b350"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:50:52.452429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.452412 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-credential-socket" pod "b4e91d97-e0c2-4406-b78a-e4701515b350" (UID: "b4e91d97-e0c2-4406-b78a-e4701515b350"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:50:52.454983 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.454940 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-envoy" pod "b4e91d97-e0c2-4406-b78a-e4701515b350" (UID: "b4e91d97-e0c2-4406-b78a-e4701515b350"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:50:52.456210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.456184 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-kube-api-access-p4h2m" pod "b4e91d97-e0c2-4406-b78a-e4701515b350" (UID: "b4e91d97-e0c2-4406-b78a-e4701515b350"). InnerVolumeSpecName "kube-api-access-p4h2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:50:52.459923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.459888 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-token" pod "b4e91d97-e0c2-4406-b78a-e4701515b350" (UID: "b4e91d97-e0c2-4406-b78a-e4701515b350"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:50:52.461255 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.461209 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-podinfo" pod "b4e91d97-e0c2-4406-b78a-e4701515b350" (UID: "b4e91d97-e0c2-4406-b78a-e4701515b350"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:50:52.550302 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.550216 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:50:52.550302 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.550253 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e91d97-e0c2-4406-b78a-e4701515b350-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:50:52.550302 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.550266 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:50:52.550302 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.550280 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4h2m\" (UniqueName: \"kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-kube-api-access-p4h2m\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:50:52.550302 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.550298 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:50:52.550674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.550312 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:50:52.550674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.550324 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:50:52.550674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.550337 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e91d97-e0c2-4406-b78a-e4701515b350-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:50:52.550674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:52.550348 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e91d97-e0c2-4406-b78a-e4701515b350-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:50:53.056992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:53.056931 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" event={"ID":"b4e91d97-e0c2-4406-b78a-e4701515b350","Type":"ContainerDied","Data":"a83df5f3156facac9cf6c543a5af559e2163833934db553c0dcc7c082b20a158"} Oct 30 23:50:53.056992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:53.056980 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs" Oct 30 23:50:53.370507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:53.057005 7570 scope.go:122] "RemoveContainer" containerID="d8b2cd330f19776fc3ffc3dadda62011a670aae92fc5df4f7240fcfd3c3777f4" Oct 30 23:50:53.370507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:53.087063 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs"] Oct 30 23:50:53.370507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:53.093086 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-j2xrs"] Oct 30 23:50:54.068416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:54.068384 7570 generic.go:356] "Generic (PLEG): container finished" podID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerID="5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870" exitCode=0 Oct 30 23:50:54.068629 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:54.068411 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" event={"ID":"f3856a78-100e-48e0-aa1e-9a04bb10c99d","Type":"ContainerDied","Data":"5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870"} Oct 30 23:50:54.068894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:54.068879 7570 scope.go:122] "RemoveContainer" containerID="5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870" Oct 30 23:50:54.070868 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:54.070838 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:50:54.070985 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:54.070926 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:50:54.070985 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:54.070978 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:50:54.610098 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:54.610063 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" path="/var/lib/kubelet/pods/b4e91d97-e0c2-4406-b78a-e4701515b350/volumes" Oct 30 23:50:54.943692 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:54.943606 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:50:54.943692 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:54.943671 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:50:55.134807 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:55.134762 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" event={"ID":"f3856a78-100e-48e0-aa1e-9a04bb10c99d","Type":"ContainerStarted","Data":"ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920"} Oct 30 23:50:55.590449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:55.589824 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:50:55.590449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:55.589945 7570 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 23:50:55.590449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:55.589996 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 23:50:56.163675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:56.163638 7570 generic.go:356] "Generic (PLEG): container finished" podID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerID="6240555c4a0f290fefac4ba05d496628c7c52e564af57f302251937420a0f092" exitCode=0 Oct 30 23:50:56.164404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:56.164267 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" event={"ID":"f179a0a0-c464-4a8b-b6cc-04a21610a832","Type":"ContainerDied","Data":"6240555c4a0f290fefac4ba05d496628c7c52e564af57f302251937420a0f092"} Oct 30 23:50:56.166112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:56.166056 7570 scope.go:122] "RemoveContainer" containerID="6240555c4a0f290fefac4ba05d496628c7c52e564af57f302251937420a0f092" Oct 30 23:50:56.168996 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:56.168960 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:50:56.169119 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:56.169056 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:50:56.169119 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:56.169091 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:50:56.589864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:56.589778 7570 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 23:50:56.590113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:56.590083 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 23:50:57.180225 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:57.180185 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" event={"ID":"f179a0a0-c464-4a8b-b6cc-04a21610a832","Type":"ContainerStarted","Data":"29f5ecced49b3f5128454a5896504d057362df6f345169fd68da0bbb5fa23782"} Oct 30 23:50:57.257507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:57.255733 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:50:57.257507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:57.256560 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 23:50:57.257507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:57.256613 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 23:50:57.600489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:57.594550 7570 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 23:50:57.600489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:57.594609 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 23:50:57.878602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:57.878526 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:50:58.255072 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:58.254990 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 23:50:58.255072 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:58.255043 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 23:50:58.610489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:58.609574 7570 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 23:50:58.610489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:58.609645 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 23:50:59.254775 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:59.254745 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 23:50:59.254957 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:59.254797 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 23:50:59.460429 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:50:59.460383 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:50:59.590256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:59.589780 7570 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 23:50:59.590256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:50:59.589885 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 23:51:00.255077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.255042 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 23:51:00.255288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.255106 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 23:51:00.621973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.621948 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:51:00.622398 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.622223 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:51:00.623063 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.623048 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:51:00.643947 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.643912 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9"] Oct 30 23:51:00.644845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.644814 7570 cpu_manager.go:414] "containerMap: removing container" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" Oct 30 23:51:00.644845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.644837 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" Oct 30 23:51:00.645053 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.644865 7570 cpu_manager.go:414] "containerMap: removing container" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" Oct 30 23:51:00.645053 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.644873 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" Oct 30 23:51:00.645916 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.645105 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" Oct 30 23:51:00.645916 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.645125 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e91d97-e0c2-4406-b78a-e4701515b350" containerName="istio-proxy" Oct 30 23:51:00.649146 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.649130 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.657762 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.657746 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2ccd42-openshift-default-dockercfg-qcc6m\"" Oct 30 23:51:00.682287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.682260 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9"] Oct 30 23:51:00.746857 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.746824 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-data\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.747065 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.746983 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-credential-socket\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.747179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.747160 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-podinfo\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.747248 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.747213 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-token\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.747248 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.747241 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-certs\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.747369 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.747313 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istiod-ca-cert\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.747369 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.747363 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-envoy\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.747501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.747443 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-socket\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.747573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.747507 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfswb\" (UniqueName: \"kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-kube-api-access-nfswb\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.848732 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.848697 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-token\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.848938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.848750 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-certs\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.848938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.848830 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istiod-ca-cert\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.848938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.848882 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-envoy\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.849393 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.848965 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-socket\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.849393 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.849002 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfswb\" (UniqueName: \"kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-kube-api-access-nfswb\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.849393 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.849042 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-data\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.849393 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.849173 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-credential-socket\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.849393 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.849312 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-socket\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.849665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.849401 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-podinfo\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.849665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.849547 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-data\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.849665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.849566 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-credential-socket\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.849810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.849789 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istiod-ca-cert\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.850015 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.849995 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-certs\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.851711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.851692 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-envoy\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.852112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.852093 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-podinfo\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.861150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.861125 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-token\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.861233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.861159 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfswb\" (UniqueName: \"kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-kube-api-access-nfswb\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:00.967599 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:00.963721 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:01.257771 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:01.254612 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 23:51:01.257771 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:01.254672 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 23:51:01.519225 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:01.519151 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9"] Oct 30 23:51:01.523558 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:51:01.519421 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fc6c4a_5ced_4748_9d6f_c7cede8c7dc7.slice/crio-78a783eba81c05eca8bc4b8264e20b1cc37a37d9c8b5ac15d7b0ba42c99568c5 WatchSource:0}: Error finding container 78a783eba81c05eca8bc4b8264e20b1cc37a37d9c8b5ac15d7b0ba42c99568c5: Status 404 returned error can't find the container with id 78a783eba81c05eca8bc4b8264e20b1cc37a37d9c8b5ac15d7b0ba42c99568c5 Oct 30 23:51:01.523558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:01.523296 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:51:01.523558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:01.523409 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:51:01.523558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:01.523454 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:51:02.255310 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:02.255260 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 23:51:02.255883 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:02.255335 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 23:51:02.285391 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:02.285352 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" event={"ID":"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7","Type":"ContainerStarted","Data":"a484bb709e0b21eb10b3c20ec1ffbc432be3950bc50ca72732a27baf0121f5d4"} Oct 30 23:51:02.285748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:02.285725 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" event={"ID":"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7","Type":"ContainerStarted","Data":"78a783eba81c05eca8bc4b8264e20b1cc37a37d9c8b5ac15d7b0ba42c99568c5"} Oct 30 23:51:02.328646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:02.328512 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podStartSLOduration=2.328493464 podStartE2EDuration="2.328493464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:02.325264471 +0000 UTC m=+1873.844265561" watchObservedRunningTime="2026-10-30 23:51:02.328493464 +0000 UTC m=+1873.847494561" Oct 30 23:51:02.964442 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:02.964414 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:02.965752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:02.965729 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 23:51:02.965895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:02.965772 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 23:51:03.260594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:03.260490 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:51:03.261601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:03.261575 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:51:03.975506 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:03.969840 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 23:51:03.975506 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:03.969912 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 23:51:04.964760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:04.964723 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 23:51:04.965184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:04.964787 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 23:51:05.964609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:05.964571 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 23:51:05.964826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:05.964635 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 23:51:06.964714 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:06.964677 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 23:51:06.964916 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:06.964737 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 23:51:07.964680 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:07.964646 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 23:51:07.964893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:07.964707 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 23:51:08.970294 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:08.970269 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:08.970713 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:08.970697 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:08.971421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:08.971400 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:51:24.898524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:24.886850 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:51:25.078488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.015664 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:25.078488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.059558 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:25.078488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.059958 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:51:25.149665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.149531 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:25.193624 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.193585 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:25.280585 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.280540 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:25.387530 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.386644 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:25.434920 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.434807 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:25.457887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.457853 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:25.469584 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.469550 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:25.773541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.768671 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:25.890488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.889753 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:25.904872 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:25.902577 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:51:26.060099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:26.060008 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:26.072412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:26.072381 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:51:26.089508 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:26.073580 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:26.310550 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:26.302670 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:51:26.334025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:26.334000 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:51:26.340634 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:26.340593 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:26.341432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:26.341413 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:51:26.367485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:26.367144 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:51:29.604552 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:51:29.603447 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:51:32.561023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.560977 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx"] Oct 30 23:51:32.585528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.585218 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx"] Oct 30 23:51:32.600226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.600201 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.640355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.640325 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2b4009-openshift-default-dockercfg-mnjpt\"" Oct 30 23:51:32.750451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.750414 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-istio-token\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.750663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.750598 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-socket\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.750663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.750633 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-envoy\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.750807 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.750780 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-certs\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.750867 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.750849 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-data\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.750920 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.750906 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f68436c1-d900-4063-b853-cb9be78ceae6-istio-podinfo\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.751058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.751039 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-credential-socket\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.751202 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.751186 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f68436c1-d900-4063-b853-cb9be78ceae6-istiod-ca-cert\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.751283 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.751265 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kckmj\" (UniqueName: \"kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-kube-api-access-kckmj\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.852754 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.852712 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-istio-token\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.878727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.852802 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-socket\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.878727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.852823 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-envoy\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.878727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.852974 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-certs\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.878727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.853085 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-data\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.878727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.853162 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f68436c1-d900-4063-b853-cb9be78ceae6-istio-podinfo\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.878727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.853244 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-credential-socket\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.878727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.853307 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f68436c1-d900-4063-b853-cb9be78ceae6-istiod-ca-cert\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.878727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.853364 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kckmj\" (UniqueName: \"kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-kube-api-access-kckmj\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.911691 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.893050 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-socket\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.911691 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.893127 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-certs\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.925580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.925544 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-data\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.925715 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.925637 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-credential-socket\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.925950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.925926 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f68436c1-d900-4063-b853-cb9be78ceae6-istiod-ca-cert\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.994900 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.994826 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-envoy\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.995132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.994993 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-istio-token\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:32.995376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:32.995313 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kckmj\" (UniqueName: \"kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-kube-api-access-kckmj\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:33.006565 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:33.006378 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f68436c1-d900-4063-b853-cb9be78ceae6-istio-podinfo\") pod \"e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:33.257295 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:33.257216 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:34.246228 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:34.246196 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx"] Oct 30 23:51:34.286056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:34.286012 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:51:34.286158 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:34.286110 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:51:34.286202 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:34.286152 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:51:34.561190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:34.561117 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" event={"ID":"f68436c1-d900-4063-b853-cb9be78ceae6","Type":"ContainerStarted","Data":"c30617db6774365f2605f645b02abd554fc4a67999922b27bc2763f1ad73c11a"} Oct 30 23:51:34.561190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:34.561174 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" event={"ID":"f68436c1-d900-4063-b853-cb9be78ceae6","Type":"ContainerStarted","Data":"ac32738d430a6d0e77a64d5fd40c9f6152a014dfe151d4b64388ce9511468003"} Oct 30 23:51:34.588785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:34.588738 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" podStartSLOduration=2.588717828 podStartE2EDuration="2.588717828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:34.586697873 +0000 UTC m=+1906.105698964" watchObservedRunningTime="2026-10-30 23:51:34.588717828 +0000 UTC m=+1906.107718983" Oct 30 23:51:35.258241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:35.258208 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:35.279242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:35.265168 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:35.570672 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:35.570595 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:35.635838 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:35.571668 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:51:38.152045 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:38.152012 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2"] Oct 30 23:51:38.277654 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:38.277623 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2"] Oct 30 23:51:38.277843 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:38.277771 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:51:38.309337 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:38.284013 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2b4009\"" Oct 30 23:51:38.372737 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:38.365263 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2\" (UID: \"b12dc177-11a1-40b0-9fdf-75993ab176e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:51:38.388530 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:38.373242 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xznl4\" (UniqueName: \"kubernetes.io/projected/b12dc177-11a1-40b0-9fdf-75993ab176e6-kube-api-access-xznl4\") pod \"payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2\" (UID: \"b12dc177-11a1-40b0-9fdf-75993ab176e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:51:38.491538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:38.475668 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2\" (UID: \"b12dc177-11a1-40b0-9fdf-75993ab176e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:51:38.495744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:38.495718 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xznl4\" (UniqueName: \"kubernetes.io/projected/b12dc177-11a1-40b0-9fdf-75993ab176e6-kube-api-access-xznl4\") pod \"payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2\" (UID: \"b12dc177-11a1-40b0-9fdf-75993ab176e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:51:38.508084 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:38.493716 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2\" (UID: \"b12dc177-11a1-40b0-9fdf-75993ab176e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:51:38.596135 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:38.596100 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xznl4\" (UniqueName: \"kubernetes.io/projected/b12dc177-11a1-40b0-9fdf-75993ab176e6-kube-api-access-xznl4\") pod \"payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2\" (UID: \"b12dc177-11a1-40b0-9fdf-75993ab176e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:51:38.703527 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:38.703497 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:51:39.182090 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:39.141529 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2"] Oct 30 23:51:39.182090 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:51:39.141958 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12dc177_11a1_40b0_9fdf_75993ab176e6.slice/crio-a93e97b08d4fc30ddc8c46d8359c87022076209830fa3c0fa007e37b7a8f89c5 WatchSource:0}: Error finding container a93e97b08d4fc30ddc8c46d8359c87022076209830fa3c0fa007e37b7a8f89c5: Status 404 returned error can't find the container with id a93e97b08d4fc30ddc8c46d8359c87022076209830fa3c0fa007e37b7a8f89c5 Oct 30 23:51:39.716931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:39.716897 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" event={"ID":"b12dc177-11a1-40b0-9fdf-75993ab176e6","Type":"ContainerStarted","Data":"a93e97b08d4fc30ddc8c46d8359c87022076209830fa3c0fa007e37b7a8f89c5"} Oct 30 23:51:40.735818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:40.735777 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" event={"ID":"b12dc177-11a1-40b0-9fdf-75993ab176e6","Type":"ContainerStarted","Data":"a2397d1a972d4c1e20267b88f3965c58ed65da73260e8e63c61ae66c513cbdab"} Oct 30 23:51:40.736378 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:40.735904 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:51:40.751089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:40.751046 7570 patch_prober.go:29] interesting pod/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:51:40.751240 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:40.751126 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:51:40.761314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:40.761257 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" podStartSLOduration=2.76124159 podStartE2EDuration="2.76124159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:40.757893627 +0000 UTC m=+1912.276894719" watchObservedRunningTime="2026-10-30 23:51:40.76124159 +0000 UTC m=+1912.280242683" Oct 30 23:51:45.801398 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:45.801312 7570 generic.go:356] "Generic (PLEG): container finished" podID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerID="de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb" exitCode=0 Oct 30 23:51:45.801398 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:45.801374 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" event={"ID":"84a7842f-6043-4b93-ab32-d25e2ba8c5bb","Type":"ContainerDied","Data":"de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb"} Oct 30 23:51:45.803123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:45.801845 7570 scope.go:122] "RemoveContainer" containerID="de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb" Oct 30 23:51:45.803981 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:45.803756 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:51:45.803981 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:45.803821 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:51:45.803981 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:45.803853 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:51:46.858486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:46.857709 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" event={"ID":"84a7842f-6043-4b93-ab32-d25e2ba8c5bb","Type":"ContainerStarted","Data":"80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796"} Oct 30 23:51:47.677299 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:47.677259 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:51:47.677629 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:47.677605 7570 patch_prober.go:29] interesting pod/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 23:51:47.677685 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:47.677661 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 23:51:48.677918 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:48.677889 7570 patch_prober.go:29] interesting pod/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 23:51:48.678301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:48.677937 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 23:51:49.677753 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.677716 7570 patch_prober.go:29] interesting pod/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 23:51:49.677960 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.677779 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 23:51:49.816081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.816038 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr"] Oct 30 23:51:49.821806 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.821779 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:49.823812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.823787 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr"] Oct 30 23:51:49.826283 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.826264 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f026c1-openshift-default-dockercfg-6hgm8\"" Oct 30 23:51:49.962151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.962053 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-podinfo\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:49.962336 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.962247 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-credential-socket\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:49.962412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.962396 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-socket\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:49.963394 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.962544 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-data\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:49.963394 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.962684 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-certs\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:49.963394 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.962744 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpzk6\" (UniqueName: \"kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-kube-api-access-vpzk6\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:49.963394 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.962893 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-token\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:49.963394 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.962919 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-envoy\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:49.963394 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:49.962973 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1017ca7a-a16e-44bd-af55-d8babaa94e01-istiod-ca-cert\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.064230 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.064197 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-certs\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.064397 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.064272 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpzk6\" (UniqueName: \"kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-kube-api-access-vpzk6\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.064397 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.064361 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-token\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.064397 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.064383 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-envoy\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.064541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.064405 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1017ca7a-a16e-44bd-af55-d8babaa94e01-istiod-ca-cert\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.064541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.064474 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-podinfo\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.064605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.064542 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-credential-socket\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.064605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.064582 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-socket\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.064842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.064827 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-data\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.068984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.068958 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-certs\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.069546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.069212 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1017ca7a-a16e-44bd-af55-d8babaa94e01-istiod-ca-cert\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.069546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.069354 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-credential-socket\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.069546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.069401 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-socket\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.069950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.069937 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-envoy\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.069993 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.069937 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-data\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.070261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.070240 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-podinfo\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.080711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.080687 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-token\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.082701 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.082664 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpzk6\" (UniqueName: \"kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-kube-api-access-vpzk6\") pod \"e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.140881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.140770 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:50.479518 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:51:50.479483 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1017ca7a_a16e_44bd_af55_d8babaa94e01.slice/crio-3e05a8a2136a85be0af92165b28087911cd43e23940b0abc5b811e21c627e267 WatchSource:0}: Error finding container 3e05a8a2136a85be0af92165b28087911cd43e23940b0abc5b811e21c627e267: Status 404 returned error can't find the container with id 3e05a8a2136a85be0af92165b28087911cd43e23940b0abc5b811e21c627e267 Oct 30 23:51:50.480702 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.480676 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr"] Oct 30 23:51:50.482817 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.482783 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:51:50.483411 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.482867 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:51:50.483411 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.482895 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:51:50.683222 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.683186 7570 patch_prober.go:29] interesting pod/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 23:51:50.684205 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.684179 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 23:51:50.919238 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.919200 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" event={"ID":"1017ca7a-a16e-44bd-af55-d8babaa94e01","Type":"ContainerStarted","Data":"0bde901d48fa5140c259012bee5dd70f0ef26c4ba298b70a93e1e9c3ff2d7acd"} Oct 30 23:51:50.919607 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.919582 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" event={"ID":"1017ca7a-a16e-44bd-af55-d8babaa94e01","Type":"ContainerStarted","Data":"3e05a8a2136a85be0af92165b28087911cd43e23940b0abc5b811e21c627e267"} Oct 30 23:51:50.945428 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:50.945379 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podStartSLOduration=1.945366431 podStartE2EDuration="1.945366431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:50.943480934 +0000 UTC m=+1922.462482022" watchObservedRunningTime="2026-10-30 23:51:50.945366431 +0000 UTC m=+1922.464367519" Oct 30 23:51:51.141200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:51.141170 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:51.142401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:51.142366 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 23:51:51.142576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:51.142428 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 23:51:51.358837 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:51.358812 7570 scope.go:122] "RemoveContainer" containerID="6de559f667ef5c2a52063d30974b72ef2887a4c8b1d123df065ef8781c9de399" Oct 30 23:51:51.677888 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:51.677805 7570 patch_prober.go:29] interesting pod/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 23:51:51.677888 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:51.677861 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 23:51:51.749321 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:51.749291 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:51:52.141897 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:52.141856 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 23:51:52.142824 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:52.141925 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 23:51:52.682302 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:52.682271 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:51:52.683162 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:52.682758 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:51:52.683395 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:52.683378 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:51:53.141974 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:53.141933 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 23:51:53.142373 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:53.141999 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 23:51:54.141288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:54.141251 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 23:51:54.141503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:54.141306 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 23:51:55.141534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:55.141495 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 23:51:55.142020 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:55.141559 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 23:51:56.144494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:56.143837 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 23:51:56.144494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:56.143894 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 23:51:57.141534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:57.141454 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 23:51:57.141711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:57.141567 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 23:51:58.154025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:58.153994 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:58.154535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:58.154320 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:58.156574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:51:58.156549 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:51:59.436425 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:51:59.436387 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:52:07.423392 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.423359 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj"] Oct 30 23:52:07.430894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.430871 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.437568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.437542 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd"] Oct 30 23:52:07.444725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.444703 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.445120 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.445096 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj"] Oct 30 23:52:07.460174 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.457562 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd"] Oct 30 23:52:07.528797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.528774 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7b0299-openshift-default-dockercfg-cq9kk\"" Oct 30 23:52:07.576538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.576505 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.576669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.576582 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.576669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.576625 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.576669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.576656 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.576851 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.576762 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.577099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577081 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2lm\" (UniqueName: \"kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-kube-api-access-kz2lm\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.577156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577140 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.577261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577245 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.577322 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577294 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.577376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577341 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.577435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577417 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.577496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577456 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmfq5\" (UniqueName: \"kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-kube-api-access-dmfq5\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.577559 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577525 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.577600 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577566 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.577645 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577605 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.577734 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577718 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.577849 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577747 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.577849 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.577766 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.679291 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679228 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.679291 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679272 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.679581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679304 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.679581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679415 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.679581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679441 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.679581 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679548 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2lm\" (UniqueName: \"kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-kube-api-access-kz2lm\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679624 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679735 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679788 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679813 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679865 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679897 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmfq5\" (UniqueName: \"kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-kube-api-access-dmfq5\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679951 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.679995 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.680028 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.680321 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.680327 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.680361 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.680399 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.680427 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.680844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.680658 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.684211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.680910 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.684211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.681496 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.684211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.681530 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.684211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.682432 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.684211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.682441 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.684211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.682416 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.684211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.682657 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.684211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.682745 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.684211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.682770 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.684211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.682848 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.684211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.684017 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.691999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.691979 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2lm\" (UniqueName: \"kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-kube-api-access-kz2lm\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.692755 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.692725 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.693170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.693153 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmfq5\" (UniqueName: \"kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-kube-api-access-dmfq5\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:07.694023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.694006 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.775899 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.775878 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:07.786390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:07.786368 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:08.079854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:08.079827 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj"] Oct 30 23:52:08.080059 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:52:08.080037 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7f3b9a_978c_49f2_a52a_5293dfb1924f.slice/crio-f4ffedd06bba0b89cf93f59d2c4d3f40407db636de38c9fdaa74b9a6589ac587 WatchSource:0}: Error finding container f4ffedd06bba0b89cf93f59d2c4d3f40407db636de38c9fdaa74b9a6589ac587: Status 404 returned error can't find the container with id f4ffedd06bba0b89cf93f59d2c4d3f40407db636de38c9fdaa74b9a6589ac587 Oct 30 23:52:08.082865 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:08.082829 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:08.082982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:08.082918 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:08.082982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:08.082962 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:08.116960 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:08.116935 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd"] Oct 30 23:52:08.117973 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:52:08.117946 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf9885e_6fe7_4c33_b91a_60b11bf4b730.slice/crio-ba6cb2374b8020117f144a68b9dc8fbe6cfa9f0dd1dc298dc4291f95a13c306e WatchSource:0}: Error finding container ba6cb2374b8020117f144a68b9dc8fbe6cfa9f0dd1dc298dc4291f95a13c306e: Status 404 returned error can't find the container with id ba6cb2374b8020117f144a68b9dc8fbe6cfa9f0dd1dc298dc4291f95a13c306e Oct 30 23:52:08.121089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:08.121055 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:08.121170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:08.121122 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:08.121170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:08.121154 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:08.134435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:08.134413 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" event={"ID":"4c7f3b9a-978c-49f2-a52a-5293dfb1924f","Type":"ContainerStarted","Data":"f4ffedd06bba0b89cf93f59d2c4d3f40407db636de38c9fdaa74b9a6589ac587"} Oct 30 23:52:08.135942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:08.135913 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" event={"ID":"cbf9885e-6fe7-4c33-b91a-60b11bf4b730","Type":"ContainerStarted","Data":"ba6cb2374b8020117f144a68b9dc8fbe6cfa9f0dd1dc298dc4291f95a13c306e"} Oct 30 23:52:09.162303 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:09.162267 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" event={"ID":"4c7f3b9a-978c-49f2-a52a-5293dfb1924f","Type":"ContainerStarted","Data":"041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931"} Oct 30 23:52:09.163987 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:09.163963 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" event={"ID":"cbf9885e-6fe7-4c33-b91a-60b11bf4b730","Type":"ContainerStarted","Data":"bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf"} Oct 30 23:52:09.212495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:09.211497 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" podStartSLOduration=2.211456148 podStartE2EDuration="2.211456148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:09.18572258 +0000 UTC m=+1940.704723669" watchObservedRunningTime="2026-10-30 23:52:09.211456148 +0000 UTC m=+1940.730457239" Oct 30 23:52:09.259332 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:09.258320 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" podStartSLOduration=2.25830161 podStartE2EDuration="2.25830161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:09.231065775 +0000 UTC m=+1940.750066863" watchObservedRunningTime="2026-10-30 23:52:09.25830161 +0000 UTC m=+1940.777302700" Oct 30 23:52:09.782487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:09.777014 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:09.782487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:09.779628 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 23:52:09.782487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:09.779677 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 23:52:09.801488 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:09.790157 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:09.823487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:09.804850 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 23:52:09.823487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:09.804904 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 23:52:10.776647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:10.776614 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 23:52:10.777107 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:10.776672 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 23:52:10.810611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:10.810567 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 23:52:10.810877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:10.810849 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 23:52:11.777039 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:11.777003 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 23:52:11.777479 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:11.777049 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 23:52:11.807486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:11.804140 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 23:52:11.807486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:11.804189 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 23:52:12.776535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:12.776505 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 23:52:12.776792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:12.776557 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 23:52:12.826487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:12.808440 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 23:52:12.826487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:12.808528 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 23:52:13.776349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:13.776293 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 23:52:13.776651 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:13.776365 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 23:52:13.807486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:13.787905 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 23:52:13.807486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:13.787974 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 23:52:14.782656 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:14.782629 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:14.783068 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:14.782916 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:14.783580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:14.783550 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:52:14.791696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:14.791674 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:14.791947 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:14.791930 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:14.792677 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:14.792661 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:52:23.605499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.605432 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk"] Oct 30 23:52:23.611154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.611129 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.620013 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.617688 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-df3b5f-openshift-default-dockercfg-hmtcd\"" Oct 30 23:52:23.625449 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.625420 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk"] Oct 30 23:52:23.711103 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.711065 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-podinfo\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.711311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.711205 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qgh\" (UniqueName: \"kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-kube-api-access-82qgh\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.711311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.711258 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-data\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.711515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.711455 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8235a15-af1b-4332-ac83-1aef5bfb1286-istiod-ca-cert\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.711593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.711544 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-credential-socket\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.711593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.711570 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-certs\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.711719 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.711595 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-envoy\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.711719 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.711711 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-socket\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.711913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.711896 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-token\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.830487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.826192 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-token\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.830487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.826321 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-podinfo\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.830487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.826393 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qgh\" (UniqueName: \"kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-kube-api-access-82qgh\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.830487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.826440 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-data\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.830487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.826595 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8235a15-af1b-4332-ac83-1aef5bfb1286-istiod-ca-cert\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.830487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.826652 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-credential-socket\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.830487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.826675 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-certs\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.830487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.826699 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-envoy\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.830487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.826803 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-socket\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.830487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.827382 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-socket\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.830487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.828666 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8235a15-af1b-4332-ac83-1aef5bfb1286-istiod-ca-cert\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.854318 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.852955 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-data\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.854792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.854767 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-certs\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.855959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.855858 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-podinfo\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.856223 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.856172 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-credential-socket\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.870748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.870691 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-envoy\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.904599 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.904531 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-token\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.909021 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.908993 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qgh\" (UniqueName: \"kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-kube-api-access-82qgh\") pod \"e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:23.935697 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:23.935311 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:24.367594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:24.361451 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk"] Oct 30 23:52:24.378948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:24.378892 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:24.379214 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:24.379184 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:24.379451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:24.379428 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:25.403005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:25.402962 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" event={"ID":"d8235a15-af1b-4332-ac83-1aef5bfb1286","Type":"ContainerStarted","Data":"384a06f41c709dbd8e0b9f506ff9f630aa2e0aa1e4523c8ed6cabb86121c1c89"} Oct 30 23:52:25.403568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:25.403067 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" event={"ID":"d8235a15-af1b-4332-ac83-1aef5bfb1286","Type":"ContainerStarted","Data":"054e2ef4159b116b3a8a75ec865d77186663a5477768c85d5fe580610f584c79"} Oct 30 23:52:25.458041 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:25.457965 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" podStartSLOduration=2.457947657 podStartE2EDuration="2.457947657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:25.452030827 +0000 UTC m=+1956.971031917" watchObservedRunningTime="2026-10-30 23:52:25.457947657 +0000 UTC m=+1956.976948746" Oct 30 23:52:25.945984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:25.945955 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:25.952934 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:25.952910 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:26.411893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:26.411866 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:26.412923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:26.412904 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:52:29.205679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:29.205647 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c"] Oct 30 23:52:29.220935 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:29.220877 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:52:29.226038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:29.226008 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c"] Oct 30 23:52:29.229655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:29.229632 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-df3b5f-dockercfg-rnjbv\"" Oct 30 23:52:29.236001 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:29.235968 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c\" (UID: \"e9c9e5ed-908c-463d-8f4f-395a80ababdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:52:29.236271 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:29.236248 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snm8f\" (UniqueName: \"kubernetes.io/projected/e9c9e5ed-908c-463d-8f4f-395a80ababdd-kube-api-access-snm8f\") pod \"payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c\" (UID: \"e9c9e5ed-908c-463d-8f4f-395a80ababdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:52:29.307485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:29.284708 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-df3b5f\"" Oct 30 23:52:29.393490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:29.392276 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c\" (UID: \"e9c9e5ed-908c-463d-8f4f-395a80ababdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:52:29.393490 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:29.392674 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snm8f\" (UniqueName: \"kubernetes.io/projected/e9c9e5ed-908c-463d-8f4f-395a80ababdd-kube-api-access-snm8f\") pod \"payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c\" (UID: \"e9c9e5ed-908c-463d-8f4f-395a80ababdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:52:29.553582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:29.512872 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c\" (UID: \"e9c9e5ed-908c-463d-8f4f-395a80ababdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:52:29.564924 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:29.564886 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snm8f\" (UniqueName: \"kubernetes.io/projected/e9c9e5ed-908c-463d-8f4f-395a80ababdd-kube-api-access-snm8f\") pod \"payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c\" (UID: \"e9c9e5ed-908c-463d-8f4f-395a80ababdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:52:29.577849 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:52:29.577806 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:52:29.850636 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:29.850609 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:52:30.144255 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:30.144219 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c"] Oct 30 23:52:30.144872 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:52:30.144848 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c9e5ed_908c_463d_8f4f_395a80ababdd.slice/crio-b148fbb0cc25ede32e927038e6425249f74bfc965590f261c078e6de63cf1bd2 WatchSource:0}: Error finding container b148fbb0cc25ede32e927038e6425249f74bfc965590f261c078e6de63cf1bd2: Status 404 returned error can't find the container with id b148fbb0cc25ede32e927038e6425249f74bfc965590f261c078e6de63cf1bd2 Oct 30 23:52:30.543573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:30.543453 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" event={"ID":"e9c9e5ed-908c-463d-8f4f-395a80ababdd","Type":"ContainerStarted","Data":"437b364aa15062825ed4d26c6d17ec8b15ba4dd0916980db86c3f0f71ca026e9"} Oct 30 23:52:30.543573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:30.543540 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" event={"ID":"e9c9e5ed-908c-463d-8f4f-395a80ababdd","Type":"ContainerStarted","Data":"b148fbb0cc25ede32e927038e6425249f74bfc965590f261c078e6de63cf1bd2"} Oct 30 23:52:30.543573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:30.543569 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:52:30.563234 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:30.563183 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podStartSLOduration=1.563172037 podStartE2EDuration="1.563172037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:30.561690899 +0000 UTC m=+1962.080691990" watchObservedRunningTime="2026-10-30 23:52:30.563172037 +0000 UTC m=+1962.082173125" Oct 30 23:52:37.520115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.520072 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr"] Oct 30 23:52:37.587705 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.583788 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.664658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.658117 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp"] Oct 30 23:52:37.805885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.770414 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.805885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.770508 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.805885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.770746 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.805885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.770803 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbd7c\" (UniqueName: \"kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-kube-api-access-xbd7c\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.805885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.770941 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.805885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.770973 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.805885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.771052 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.805885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.771107 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.805885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.771157 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.858760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.857120 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:37.877830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.877302 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.877830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.877362 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbd7c\" (UniqueName: \"kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-kube-api-access-xbd7c\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.877830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.877504 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.877830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.877541 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.877830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.877615 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.877830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.877668 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.877830 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.877713 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.878417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.877858 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.878417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.877909 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.913487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.906116 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.923496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.922258 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.923496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.922958 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.923496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.923219 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.938510 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.938352 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:37.993500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.992888 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:37.993500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.993040 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:37.993500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.993067 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:37.993500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.993165 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44hlq\" (UniqueName: \"kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-kube-api-access-44hlq\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:37.993500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.993264 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb95af9b-7595-4dd9-839a-e698e2a22c07-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:37.993500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.993321 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:37.993500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.993367 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:37.997596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.993551 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:37.997596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:37.993665 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.029481 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.025246 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:38.050107 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.031716 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr"] Oct 30 23:52:38.050107 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.048558 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:38.050107 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.049048 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:38.070055 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.069312 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbd7c\" (UniqueName: \"kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-kube-api-access-xbd7c\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:38.129994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.096311 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp"] Oct 30 23:52:38.129994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.097672 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb95af9b-7595-4dd9-839a-e698e2a22c07-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.129994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.097727 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.129994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.097771 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.129994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.097890 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.129994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.097958 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.129994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.098161 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.129994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.098256 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.129994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.098281 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.129994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.098358 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44hlq\" (UniqueName: \"kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-kube-api-access-44hlq\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.129994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.129756 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:52:38.255631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.136813 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.255631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.137720 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.255631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.138623 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.255631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.138837 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.255631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.148688 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb95af9b-7595-4dd9-839a-e698e2a22c07-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.255631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.165689 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.255631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.177938 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.255631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.254602 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44hlq\" (UniqueName: \"kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-kube-api-access-44hlq\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.257749 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.256250 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:38.468384 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:38.468351 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:52:39.357605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:39.354590 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr"] Oct 30 23:52:39.372420 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:39.365710 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp"] Oct 30 23:52:39.375322 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:52:39.371555 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b86c0c_3f2f_4f9d_be26_948cb9084373.slice/crio-d227395e13ef5d3712a842aa0cba775cedb5e82bed38de62721b43a42c08edeb WatchSource:0}: Error finding container d227395e13ef5d3712a842aa0cba775cedb5e82bed38de62721b43a42c08edeb: Status 404 returned error can't find the container with id d227395e13ef5d3712a842aa0cba775cedb5e82bed38de62721b43a42c08edeb Oct 30 23:52:39.419861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:39.419785 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:39.420006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:39.419895 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:39.420643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:39.420537 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:39.420923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:39.420886 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:39.421062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:39.421045 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:39.421180 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:39.421161 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:52:45.039662 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:45.039615 7570 trace.go:236] Trace[1961306682]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" (30-Oct-2026 23:52:41.142) (total time: 1058ms): Oct 30 23:52:45.039662 ip-10-0-137-81 kubenswrapper[7570]: Trace[1961306682]: [1.058733309s] [1.058733309s] END Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:42.669158 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:44.982830 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:44.521631 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:42.671919 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:44.039605 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:44.494752 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:44.142624 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:45.084844 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:44.995523 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:42.836703 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:44.024529 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:45.733921 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:45.971899 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.8307915s: [/var/lib/containers/storage/overlay/cd3058ba6d2cf52b3aeb9de37169925252ba5677fc23be231ae11bff86ef11a3/diff /var/log/pods/e2e-models-e2e-worker-w4-843d8c_premium-w4-843d8c-kserve-7844d66896-27fxd_4f1bda07-310d-43a8-a926-9004487b0034/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:46.037038 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:46.040197 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.980990963s: [/var/lib/containers/storage/overlay/9f074815449dfeb2c6adcd6fdde2b1517ee82a0a4d04d335711556b364004ffb/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc_12efb180-e881-40a3-9001-8a39a2ddd352/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:46.043850 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.90143281s: [/var/lib/containers/storage/overlay/3d6dbccaefb2e840fc49eb5118cfccdb9ef635473575b9dadf0c545ae40325f3/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:46.064827 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.543303683s: [/var/lib/containers/storage/overlay/e91783390aa97d397922076705ae546742cf327cb4412039da1bc9f0d0be7b8e/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws_5244915c-9573-489c-b1f8-e22948899df3/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:46.176870 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:43.274254 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:43.944036 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:46.264793 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podStartSLOduration=9.264779747 podStartE2EDuration="9.264779747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:46.264557363 +0000 UTC m=+1977.783558442" watchObservedRunningTime="2026-10-30 23:52:46.264779747 +0000 UTC m=+1977.783780832" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:46.290351 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podStartSLOduration=9.290338689 podStartE2EDuration="9.290338689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:46.28916121 +0000 UTC m=+1977.808162299" watchObservedRunningTime="2026-10-30 23:52:46.290338689 +0000 UTC m=+1977.809339778" Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:46.472347 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 23:52:46.568646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:46.472409 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 23:52:48.758132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:46.904631 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:52:48.758132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:46.907554 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-4blbf" podUID="1793bf9b-92db-4dae-bfd6-f86e5b076739" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:48.758132 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:52:47.023894 7570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="5.902s" Oct 30 23:52:48.758132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:47.023934 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" event={"ID":"cb95af9b-7595-4dd9-839a-e698e2a22c07","Type":"ContainerStarted","Data":"a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588"} Oct 30 23:52:48.758132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:47.036613 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:52:48.758132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:47.036668 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:48.758132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:47.132303 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 23:52:48.758132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:47.132364 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:50.801556 7570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-xsv62 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:50.801619 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podUID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:51.260505 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:51.260573 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:51.266790 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:51.266842 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:52:51.352862 7570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.401s" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:51.380919 7570 trace.go:236] Trace[258105620]: "Calculate volume metrics of metrics-client-ca for pod openshift-monitoring/openshift-state-metrics-7bd7489957-t49gb" (30-Oct-2026 23:52:50.329) (total time: 1051ms): Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: Trace[258105620]: [1.051499239s] [1.051499239s] END Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:51.462665 7570 trace.go:236] Trace[1033088977]: "iptables ChainExists" (30-Oct-2026 23:52:48.927) (total time: 2535ms): Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: Trace[1033088977]: [2.535037534s] [2.535037534s] END Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.161858 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.161881 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.161886 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.161969 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.163118 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.163129 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.163241 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.163271 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.163361 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.164200 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.164222 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.165269 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.165314 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.165316 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.165322 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.166385 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.181873 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.181906 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.182975 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.183003 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.183021 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.183027 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.231713 7570 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-g86k2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.231807 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" podUID="9c20fb11-2dcf-445e-96cd-4d97045a2ab9" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.15:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.234590 7570 patch_prober.go:29] interesting pod/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.234649 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.268603 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.274591 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" podUID="d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.18:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:52.311549 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.162248 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.162297 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.162264 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.162302 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.163405 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.163437 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.163443 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.163523 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.164516 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.165597 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.165632 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.183019 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.183036 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.183136 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.183171 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.183131 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.281470 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:53.312598 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.163137 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.163149 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.163162 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.164516 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.164523 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.164546 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.164595 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.164635 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.166662 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.166744 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.184071 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.231240 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.232333 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.233414 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.268547 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.290655 7570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-xsv62 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.290737 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podUID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.311798 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.312889 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.704317 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded" start-of-body= Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:54.704390 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:55.164850 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:55.167039 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:55.167072 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:55.184367 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:55.436532 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:55.651928 7570 patch_prober.go:29] interesting pod/dns-default-h4hmx container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:55.652025 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-h4hmx" podUID="cb76bb5e-c0a6-4a38-8732-40193dd39cac" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:56.182335 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": context deadline exceeded" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:56.182450 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:52:56.284655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:52:56.184681 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:53:57.498374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:21.118708 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": context deadline exceeded" Oct 30 23:53:57.498374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:22.027540 7570 patch_prober.go:29] interesting pod/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:53:57.498374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:22.027645 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" podUID="d8235a15-af1b-4332-ac83-1aef5bfb1286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:53:57.498374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:22.032231 7570 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mxggs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:53:57.498374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:22.032273 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" podUID="363febe9-9102-48c9-8faa-09bf49151bf0" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.116:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:53:57.498374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:22.033057 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": context deadline exceeded" Oct 30 23:53:57.498374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:22.046032 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": context deadline exceeded" Oct 30 23:53:57.498374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:22.051593 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": context deadline exceeded" Oct 30 23:57:44.639032 ip-10-0-137-81 kubenswrapper[7570]: rConn.createTransport failed to connect to {Addr: "10.134.0.158:50052", ServerName: "10.134.0.158:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:51722->10.134.0.158:50052: use of closed network connection" Oct 30 23:57:44.639032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.591432 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 30 23:57:44.639032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.591606 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" podUID="d387eda1-c364-4443-ba0d-619368f57f28" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.132:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.639032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.591653 7570 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.639032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.591736 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.640341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.591814 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" podUID="1d0b4116-23a4-4074-8dfb-93b7bbf53681" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.146:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.640341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.591823 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.640341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.591915 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.640341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.623847 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.640341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.623851 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" podUID="08022df4-0b83-43af-9b14-b85ab9f83be3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.145:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.641127 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:23.632615 7570 logging.go:55] [core] [Channel #2700 SubChannel #2702]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.146:50052", ServerName: "10.134.0.146:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:58198->10.134.0.146:50052: use of closed network connection" Oct 30 23:57:44.641127 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:23.632668 7570 logging.go:55] [core] [Channel #2701 SubChannel #2703]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.132:50052", ServerName: "10.134.0.132:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:39558->10.134.0.132:50052: use of closed network connection" Oct 30 23:57:44.641127 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:23.664700 7570 logging.go:55] [core] [Channel #2708 SubChannel #2712]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.145:50052", ServerName: "10.134.0.145:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:48834->10.134.0.145:50052: use of closed network connection" Oct 30 23:57:44.641127 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:23.665617 7570 logging.go:55] [core] [Channel #2709 SubChannel #2717]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.45:50052", ServerName: "10.134.0.45:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:58508->10.134.0.45:50052: use of closed network connection" Oct 30 23:57:44.641686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.691976 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.641686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.692034 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.641686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.721290 7570 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-g86k2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.641686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.721327 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" podUID="9c20fb11-2dcf-445e-96cd-4d97045a2ab9" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.15:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.641686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.735039 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" podUID="ca8613ba-3272-43d3-9661-33618f859cff" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.139:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.642144 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.736087 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" podUID="1e8a6ea5-1c91-462f-94fd-7136a48971e9" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.44:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.642144 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.742291 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.642144 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.742297 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-t7b8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.642144 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.742379 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.642144 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.742447 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.643613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.811046 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Oct 30 23:57:44.643613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.811095 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 30 23:57:44.643613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.811084 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.643613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.811400 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.643613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.811427 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.643613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.811476 7570 patch_prober.go:29] interesting pod/metrics-server-5946dc94c7-djbgv container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.14:10250/livez\": context deadline exceeded" start-of-body= Oct 30 23:57:44.643948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.811430 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.643948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.811501 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" podUID="ed360b42-1a02-4474-9bf1-2117d1b438f1" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.14:10250/livez\": context deadline exceeded" Oct 30 23:57:44.643948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.811947 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" podUID="201fbcb8-8fed-44e3-b60d-4b89d8c114ca" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.138:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.643948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.812097 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" podUID="131867e6-7957-48d7-96d5-50f9e8ff45b2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.141:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.643948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.812201 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.644193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.812228 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.644193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.823981 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.644193 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:23.824008 7570 logging.go:55] [core] [Channel #2716 SubChannel #2718]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.44:50052", ServerName: "10.134.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:39932->10.134.0.44:50052: use of closed network connection" Oct 30 23:57:44.644193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.824021 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.644193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.824162 7570 patch_prober.go:29] interesting pod/metrics-server-5946dc94c7-djbgv container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.645486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.824206 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" podUID="ed360b42-1a02-4474-9bf1-2117d1b438f1" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.645486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.824293 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 30 23:57:44.645486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.824683 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:57:44.645486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.824774 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:57:44.645486 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.825169 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" podUID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.156:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.645794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.824796 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:57:44.645794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.824838 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 23:57:44.645794 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:23.824930 7570 logging.go:55] [core] [Channel #2713 SubChannel #2725]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.139:50052", ServerName: "10.134.0.139:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:45302->10.134.0.139:50052: use of closed network connection" Oct 30 23:57:44.645794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.824822 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:57:44.645794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.825262 7570 patch_prober.go:29] interesting pod/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.646596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.825287 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.646596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.825303 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" podUID="b12dc177-11a1-40b0-9fdf-75993ab176e6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.166:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.646596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.825352 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.646596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.825426 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.646596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.825659 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 23:57:44.646907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.826248 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.646907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.826337 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.646907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.826576 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.646907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.826602 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.646907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.826632 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:57:44.647192 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.825742 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.647192 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.827685 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.647192 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.827729 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.647192 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.827809 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.647192 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.828889 7570 trace.go:236] Trace[308161420]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" (30-Oct-2026 23:53:22.760) (total time: 1067ms): Oct 30 23:57:44.647192 ip-10-0-137-81 kubenswrapper[7570]: Trace[308161420]: [1.067868243s] [1.067868243s] END Oct 30 23:57:44.647573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.829297 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.647573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.829369 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" podUID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.155:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.647573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.829829 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.647573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.829873 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.647573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.831985 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" podUID="12efb180-e881-40a3-9001-8a39a2ddd352" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.163:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.647870 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.832004 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" podUID="5244915c-9573-489c-b1f8-e22948899df3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.162:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.647870 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.832184 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" podUID="522aa71b-47e4-44b1-825c-bf81305d21b4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.41:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.647870 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.832267 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.647870 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.832299 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.647870 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.832302 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.648091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.832340 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.648091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.832372 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.648091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.832426 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.648091 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:23.853618 7570 logging.go:55] [core] [Channel #2740 SubChannel #2741]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.138:50052", ServerName: "10.134.0.138:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:46206->10.134.0.138:50052: use of closed network connection" Oct 30 23:57:44.648091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.864582 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.648301 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:23.872605 7570 logging.go:55] [core] [Channel #2750 SubChannel #2752]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.162:50052", ServerName: "10.134.0.162:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:45608->10.134.0.162:50052: use of closed network connection" Oct 30 23:57:44.648301 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:23.872618 7570 logging.go:55] [core] [Channel #2743 SubChannel #2744]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.41:50052", ServerName: "10.134.0.41:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:54758->10.134.0.41:50052: use of closed network connection" Oct 30 23:57:44.648301 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:23.872624 7570 logging.go:55] [core] [Channel #2749 SubChannel #2751]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.163:50052", ServerName: "10.134.0.163:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:45996->10.134.0.163:50052: use of closed network connection" Oct 30 23:57:44.648301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.919386 7570 trace.go:236] Trace[1376980743]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" (30-Oct-2026 23:53:22.766) (total time: 1153ms): Oct 30 23:57:44.648301 ip-10-0-137-81 kubenswrapper[7570]: Trace[1376980743]: [1.153111469s] [1.153111469s] END Oct 30 23:57:44.648301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:23.940929 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.883072581s: [/var/lib/containers/storage/overlay/01bc050ff3ede245fc5b6578e08988263fdbbae208afaf2849addc5a2ddefe4b/diff /var/log/pods/openshift-ingress_e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj_4c7f3b9a-978c-49f2-a52a-5293dfb1924f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.648573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.223083 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.153895741s: [/var/lib/containers/storage/overlay/3850d1bc8fbc7eb22abcc4a003e89aeb5ab4318b239a2b288cd6c99729884848/diff /var/log/pods/openshift-ingress_e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd_cbf9885e-6fe7-4c33-b91a-60b11bf4b730/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.648573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.233390 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded" Oct 30 23:57:44.648573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.233485 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.648573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.233513 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.648573 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.233596 7570 logging.go:55] [core] [Channel #2737 SubChannel #2738]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.141:50052", ServerName: "10.134.0.141:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:58274->10.134.0.141:50052: use of closed network connection" Oct 30 23:57:44.648778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.233804 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" podUID="7371f1ff-163e-49b2-a4ea-5a9bf3a18215" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.40:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.648778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.233833 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.648778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.233886 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" podUID="d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.18:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.648778 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.234319 7570 logging.go:55] [core] [Channel #2746 SubChannel #2747]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.156:50052", ServerName: "10.134.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:59464->10.134.0.156:50052: use of closed network connection" Oct 30 23:57:44.648778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.234345 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-4blbf" podUID="1793bf9b-92db-4dae-bfd6-f86e5b076739" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.648980 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.234427 7570 logging.go:55] [core] [Channel #2734 SubChannel #2735]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.166:50052", ServerName: "10.134.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:59384->10.134.0.166:50052: use of closed network connection" Oct 30 23:57:44.648980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.234449 7570 patch_prober.go:29] interesting pod/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.648980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.234491 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.648980 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.234756 7570 logging.go:55] [core] [Channel #2731 SubChannel #2732]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:48802->10.134.0.171:50052: use of closed network connection" Oct 30 23:57:44.648980 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.234790 7570 logging.go:55] [core] [Channel #2755 SubChannel #2757]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.155:50052", ServerName: "10.134.0.155:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:39642->10.134.0.155:50052: use of closed network connection" Oct 30 23:57:44.649172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.235361 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 23:57:44.649172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.235396 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 23:57:44.649172 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.274633 7570 logging.go:55] [core] [Channel #2727 SubChannel #2729]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.40:50052", ServerName: "10.134.0.40:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:33434->10.134.0.40:50052: use of closed network connection" Oct 30 23:57:44.649172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.327401 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" podUID="6843e367-692b-41f0-a9c6-7704d732f009" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.149:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.649172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.346268 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": context deadline exceeded" Oct 30 23:57:44.649352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.347380 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.649352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.349560 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" podUID="37d7f8cf-f311-4b98-a6b7-25a043ac78ae" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.160:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.649352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.349564 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" podUID="c070c544-43c5-4796-b385-6c4094a63624" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.42:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.649352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.350301 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" podUID="7f016eec-7091-4a3e-847e-7a382085ffef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.43:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.649352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.353413 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.649578 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.355037 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.649578 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.355074 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.649578 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.355083 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" podUID="e0f6ac75-7dd2-44be-8b54-92ff018b8a3d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.148:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.649578 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.362506 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.649578 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.362538 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.649773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.362881 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.649773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.363174 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.649773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.365857 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.649773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.365886 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" podUID="f68436c1-d900-4063-b853-cb9be78ceae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.649773 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.368641 7570 logging.go:55] [core] [Channel #2756 SubChannel #2758]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.149:50052", ServerName: "10.134.0.149:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:36714->10.134.0.149:50052: use of closed network connection" Oct 30 23:57:44.649986 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.389394 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.649986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.390621 7570 logging.go:55] [core] [Channel #2759 SubChannel #2762]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.42:50052", ServerName: "10.134.0.42:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:45410->10.134.0.42:50052: use of closed network connection" Oct 30 23:57:44.649986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.390647 7570 logging.go:55] [core] [Channel #2760 SubChannel #2763]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.160:50052", ServerName: "10.134.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:56782->10.134.0.160:50052: use of closed network connection" Oct 30 23:57:44.649986 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.391595 7570 logging.go:55] [core] [Channel #2761 SubChannel #2764]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.43:50052", ServerName: "10.134.0.43:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:53218->10.134.0.43:50052: use of closed network connection" Oct 30 23:57:44.650147 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.396603 7570 logging.go:55] [core] [Channel #2766 SubChannel #2768]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.148:50052", ServerName: "10.134.0.148:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:56788->10.134.0.148:50052: use of closed network connection" Oct 30 23:57:44.650147 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.402611 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.650147 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.402664 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.650147 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.417650 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.650147 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.417689 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.650352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.430148 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded" Oct 30 23:57:44.650352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.471865 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 23:57:44.650352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.471913 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 23:57:44.650352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.592568 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" podUID="d387eda1-c364-4443-ba0d-619368f57f28" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.132:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.650352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.592670 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.650590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.592710 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.650590 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.634619 7570 logging.go:55] [core] [Channel #2773 SubChannel #2774]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.132:50052", ServerName: "10.134.0.132:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:39090->10.134.0.132:50052: use of closed network connection" Oct 30 23:57:44.650590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.811410 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Oct 30 23:57:44.650590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.811528 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.650590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.811615 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.650791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.812497 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" podUID="201fbcb8-8fed-44e3-b60d-4b89d8c114ca" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.138:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.650791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.812514 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" podUID="131867e6-7957-48d7-96d5-50f9e8ff45b2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.141:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.650791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.812553 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.650791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.812593 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.650791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.824923 7570 patch_prober.go:29] interesting pod/metrics-server-5946dc94c7-djbgv container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.651027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.824970 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" podUID="ed360b42-1a02-4474-9bf1-2117d1b438f1" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.825114 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.651027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.825152 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.825219 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.825240 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.651232 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.825265 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651232 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.825306 7570 patch_prober.go:29] interesting pod/metrics-server-5946dc94c7-djbgv container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.651232 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.825332 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" podUID="ed360b42-1a02-4474-9bf1-2117d1b438f1" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651232 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.825554 7570 patch_prober.go:29] interesting pod/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.651232 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.825577 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651475 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.826418 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651475 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.826416 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651475 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.826741 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651475 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.827127 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651475 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.827249 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" podUID="b12dc177-11a1-40b0-9fdf-75993ab176e6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.166:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.651688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.827413 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.827648 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.827870 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.828260 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651688 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.828552 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.829545 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.651884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.829588 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.829612 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.651884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.829655 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.651884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.829663 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" podUID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.155:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.652080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.829712 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.652080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.829728 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.652080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.829979 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 23:57:44.652080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.830242 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded" Oct 30 23:57:44.652080 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.830311 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" podUID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.156:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.652288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.830898 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.652288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.830982 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.652288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.831013 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.652288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.832228 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" podUID="12efb180-e881-40a3-9001-8a39a2ddd352" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.163:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.652288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.832395 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" podUID="522aa71b-47e4-44b1-825c-bf81305d21b4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.41:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.652617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.832981 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" podUID="5244915c-9573-489c-b1f8-e22948899df3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.162:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.652617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.833040 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.652617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.833060 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.652617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.833268 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.652617 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.833291 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.652827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.833401 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.652827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.833437 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.652827 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.853634 7570 logging.go:55] [core] [Channel #2777 SubChannel #2778]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.141:50052", ServerName: "10.134.0.141:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:58286->10.134.0.141:50052: use of closed network connection" Oct 30 23:57:44.652827 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.854602 7570 logging.go:55] [core] [Channel #2776 SubChannel #2779]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.138:50052", ServerName: "10.134.0.138:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:46222->10.134.0.138:50052: use of closed network connection" Oct 30 23:57:44.652827 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.867640 7570 logging.go:55] [core] [Channel #2780 SubChannel #2783]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.166:50052", ServerName: "10.134.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:59388->10.134.0.166:50052: use of closed network connection" Oct 30 23:57:44.653024 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.870694 7570 logging.go:55] [core] [Channel #2781 SubChannel #2782]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.156:50052", ServerName: "10.134.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:59478->10.134.0.156:50052: use of closed network connection" Oct 30 23:57:44.653024 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.870706 7570 logging.go:55] [core] [Channel #2785 SubChannel #2786]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.155:50052", ServerName: "10.134.0.155:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:39646->10.134.0.155:50052: use of closed network connection" Oct 30 23:57:44.653024 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.872603 7570 logging.go:55] [core] [Channel #2784 SubChannel #2788]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:48806->10.134.0.171:50052: use of closed network connection" Oct 30 23:57:44.653024 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.872611 7570 logging.go:55] [core] [Channel #2794 SubChannel #2797]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.163:50052", ServerName: "10.134.0.163:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:46006->10.134.0.163:50052: use of closed network connection" Oct 30 23:57:44.653189 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.873608 7570 logging.go:55] [core] [Channel #2796 SubChannel #2799]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.41:50052", ServerName: "10.134.0.41:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:54764->10.134.0.41:50052: use of closed network connection" Oct 30 23:57:44.653189 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:24.873647 7570 logging.go:55] [core] [Channel #2795 SubChannel #2798]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.162:50052", ServerName: "10.134.0.162:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:45612->10.134.0.162:50052: use of closed network connection" Oct 30 23:57:44.653189 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:24.906581 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.653189 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.033337 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.653189 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.046686 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.653368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.052049 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": context deadline exceeded" Oct 30 23:57:44.653368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.070565 7570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-xsv62 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.653368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.070603 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podUID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.653368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.130632 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 23:57:44.653368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.130673 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 23:57:44.653587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.234593 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" podUID="7371f1ff-163e-49b2-a4ea-5a9bf3a18215" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.40:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.653587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.275575 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-4blbf" podUID="1793bf9b-92db-4dae-bfd6-f86e5b076739" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.653587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.275606 7570 patch_prober.go:29] interesting pod/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.653587 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:25.275626 7570 logging.go:55] [core] [Channel #2803 SubChannel #2804]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.40:50052", ServerName: "10.134.0.40:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:33444->10.134.0.40:50052: use of closed network connection" Oct 30 23:57:44.653587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.275610 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" podUID="d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.18:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.653785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.275643 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.653785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.284550 7570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-xsv62 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.653785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.284583 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podUID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.653785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.351758 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.653785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.353954 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.653967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.363645 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:57:44.653967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.363697 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.653967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.363666 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.653967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.363967 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.653967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.395599 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654161 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.395710 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 23:57:44.654161 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.471366 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 23:57:44.654161 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.471422 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 23:57:44.654161 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.477676 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654161 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.630936 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.767970 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": context deadline exceeded" Oct 30 23:57:44.654347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.770449 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.776551 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-89shx" podUID="d3eeb220-e1ba-46e2-94c8-2bb19e02dde9" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.779690 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.812478 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 30 23:57:44.654618 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.826686 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:57:44.654618 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.826713 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654618 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.827251 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654618 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.827542 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654618 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.827789 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.828049 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.828354 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.828627 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.830005 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.654826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.830009 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.655026 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.830146 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.655026 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.830512 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" podUID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.156:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.655026 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.832769 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" podUID="522aa71b-47e4-44b1-825c-bf81305d21b4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.41:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.655026 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.868389 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.107828378s: [/var/lib/containers/storage/overlay/856479fdaa0b08ec4ff27d383d0ca8e191f42ae8159e758590c5f5d05201269c/diff /var/log/pods/openshift-jobset-operator_jobset-controller-manager-57b79bd74c-mqnfk_f7c5040c-6dbc-424d-a258-94092e75a027/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.655026 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:25.871659 7570 logging.go:55] [core] [Channel #2806 SubChannel #2807]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.156:50052", ServerName: "10.134.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:59488->10.134.0.156:50052: use of closed network connection" Oct 30 23:57:44.655227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.874570 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.655227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.874600 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.655227 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:25.874613 7570 logging.go:55] [core] [Channel #2809 SubChannel #2810]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.41:50052", ServerName: "10.134.0.41:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:54768->10.134.0.41:50052: use of closed network connection" Oct 30 23:57:44.655227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.909594 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.655227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.909633 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.655476 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.942054 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" podUID="5244915c-9573-489c-b1f8-e22948899df3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.162:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.655476 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:25.942061 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" podUID="12efb180-e881-40a3-9001-8a39a2ddd352" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.163:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.655476 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:25.982608 7570 logging.go:55] [core] [Channel #2813 SubChannel #2814]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.162:50052", ServerName: "10.134.0.162:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:45626->10.134.0.162:50052: use of closed network connection" Oct 30 23:57:44.655476 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:53:25.983607 7570 logging.go:55] [core] [Channel #2812 SubChannel #2815]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.163:50052", ServerName: "10.134.0.163:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:46012->10.134.0.163:50052: use of closed network connection" Oct 30 23:57:44.655476 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.067535 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 30 23:57:44.655686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.130171 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 23:57:44.655686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.130235 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 23:57:44.655686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.296672 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.655686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.296799 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.655686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.350644 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.655892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.354810 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.655892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.364534 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:57:44.655892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.364559 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.655892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.374625 7570 patch_prober.go:29] interesting pod/dns-default-h4hmx container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.655892 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.374654 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-h4hmx" podUID="cb76bb5e-c0a6-4a38-8732-40193dd39cac" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.656092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.396521 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.656092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.471995 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 23:57:44.656092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.472032 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 23:57:44.656092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.827681 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.656092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.827711 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.656351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.827718 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:57:44.656351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.827863 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:57:44.656351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.827687 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:57:44.656351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.828169 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:57:44.656351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.828537 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:57:44.656351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.828805 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.656619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.830090 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.656619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.830214 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.656619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.830233 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.656619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.965820 7570 patch_prober.go:29] interesting pod/jobset-controller-manager-57b79bd74c-mqnfk container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.134.0.26:8081/healthz\": dial tcp 10.134.0.26:8081: connect: connection refused" start-of-body= Oct 30 23:57:44.656619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:26.965871 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" podUID="f7c5040c-6dbc-424d-a258-94092e75a027" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/healthz\": dial tcp 10.134.0.26:8081: connect: connection refused" Oct 30 23:57:44.656810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.035120 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.656810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.039957 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.656810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.066361 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.656810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.069583 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.656810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.074568 7570 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mxggs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.657010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.074606 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" podUID="363febe9-9102-48c9-8faa-09bf49151bf0" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.116:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.103646 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.130367 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 23:57:44.657010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.130408 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 23:57:44.657010 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.224678 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657202 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.355265 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 23:57:44.657202 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.365028 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:57:44.657202 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.365066 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657202 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.396772 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657202 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.452582 7570 patch_prober.go:29] interesting pod/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.657408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.452632 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" podUID="d8235a15-af1b-4332-ac83-1aef5bfb1286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.470941 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 23:57:44.657408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.470980 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 23:57:44.657408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.580509 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.789572 7570 patch_prober.go:29] interesting pod/dns-default-h4hmx container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.134.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.657643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.789617 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-h4hmx" podUID="cb76bb5e-c0a6-4a38-8732-40193dd39cac" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.812491 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.828142 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:57:44.657643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.828156 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:57:44.657643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.828178 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:57:44.657844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.828189 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.828246 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.828834 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.828868 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.657844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.829018 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.658056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.830348 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:57:44.658056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.830368 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.658056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.830365 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.658056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.841100 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": context deadline exceeded" Oct 30 23:57:44.658056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:27.857554 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-89shx" podUID="d3eeb220-e1ba-46e2-94c8-2bb19e02dde9" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.658301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:28.054555 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 30 23:57:44.658301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:54:07.018990 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": context deadline exceeded" Oct 30 23:57:44.658301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:54:09.213162 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": context deadline exceeded" Oct 30 23:57:44.658301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:54:09.213169 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": context deadline exceeded" Oct 30 23:57:44.658301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:54:12.137983 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.658301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:54:13.691650 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": context deadline exceeded" Oct 30 23:57:44.658650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:47.097142 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.658650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:48.253001 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.658650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.005283 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" podUID="7f016eec-7091-4a3e-847e-7a382085ffef" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.43:50052: i/o timeout" Oct 30 23:57:44.658650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.005720 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.45:50052: i/o timeout" Oct 30 23:57:44.658650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.054269 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" podUID="522aa71b-47e4-44b1-825c-bf81305d21b4" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.41:50052: i/o timeout" Oct 30 23:57:44.658853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.054497 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" podUID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.155:50052: i/o timeout" Oct 30 23:57:44.658853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.054664 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" podUID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.156:50052: i/o timeout" Oct 30 23:57:44.658853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:51.049552 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:57:44.658853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:54.582406 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:57:44.658853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:52.667804 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": context deadline exceeded" Oct 30 23:57:44.658853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.059429 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.659078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.117454 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.659078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:56.487173 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.659078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.471978 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.659078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.490372 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" podUID="a74a48f3-2b68-424b-9489-33e22dab2f2f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.152:50052: i/o timeout" Oct 30 23:57:44.659078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.570232 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": context deadline exceeded" Oct 30 23:57:44.659290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.571501 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": context deadline exceeded" Oct 30 23:57:44.659290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:03.413900 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" podUID="d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.18:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.659290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:03.409570 7570 trace.go:236] Trace[604943188]: "Calculate volume metrics of bound-sa-token for pod cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" (30-Oct-2026 23:53:43.690) (total time: 79718ms): Oct 30 23:57:44.659290 ip-10-0-137-81 kubenswrapper[7570]: Trace[604943188]: [1m19.718954005s] [1m19.718954005s] END Oct 30 23:57:44.659290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.571948 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" podUID="12efb180-e881-40a3-9001-8a39a2ddd352" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.163:50052: i/o timeout" Oct 30 23:57:44.659290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.771722 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.908823091s: [/var/lib/containers/storage/overlay/3d4771aeb60e85fec65b07e561834bcbadbe702e861546262a8b9466d98f9e46/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.659290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.902311 7570 patch_prober.go:29] interesting pod/jobset-controller-manager-57b79bd74c-mqnfk container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.659612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:03.471280 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.659612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.423146 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.659612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.902887 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.659612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903035 7570 patch_prober.go:29] interesting pod/jobset-controller-manager-57b79bd74c-mqnfk container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.134.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.659612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903068 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" podUID="f7c5040c-6dbc-424d-a258-94092e75a027" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.659823 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.423159 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.659823 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903124 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.659823 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.423146 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.659823 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903186 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.659823 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.423206 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.660067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903249 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.660067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.423236 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.660067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903298 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.660067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.423234 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.660067 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903359 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" podUID="f68436c1-d900-4063-b853-cb9be78ceae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.660282 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.423256 7570 patch_prober.go:29] interesting pod/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.660282 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903441 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" podUID="d8235a15-af1b-4332-ac83-1aef5bfb1286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.660282 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.423260 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.660282 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903528 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.660282 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.423441 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.660619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903610 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.660619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.785971 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.660619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.790358 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.660619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903691 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.660619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.790591 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.662254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903774 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.662254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.904081 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" podUID="f7c5040c-6dbc-424d-a258-94092e75a027" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.662254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.903670 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.662254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.792298 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-t7b8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.662254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.904513 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.662555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:04.801756 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.662555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.904582 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.662555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.005503 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.662555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.904665 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.662555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.005612 7570 trace.go:236] Trace[1939728927]: "Calculate volume metrics of istio-envoy for pod openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" (30-Oct-2026 23:54:11.559) (total time: 53446ms): Oct 30 23:57:44.662555 ip-10-0-137-81 kubenswrapper[7570]: Trace[1939728927]: [53.446280463s] [53.446280463s] END Oct 30 23:57:44.662788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.052167 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 29.320391042s: [/var/lib/containers/storage/overlay/eaa49feddcea5336a6b8866cb5fac05f1997e20cf80087ff0dcca47e53553088/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.662788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.052219 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 29.320451655s: [/var/lib/containers/storage/overlay/e7953bf176b67f03f97efb84121aa6ac183dec626827f9494323b1e22705088a/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.662788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.054972 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 29.311229537s: [/var/lib/containers/storage/overlay/85cf03ea116047624f25189ad4e954125babfdeb0f372b9e0774b38f427fcc00/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.662788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.944259 7570 trace.go:236] Trace[1633421329]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" (30-Oct-2026 23:55:04.617) (total time: 1327ms): Oct 30 23:57:44.662788 ip-10-0-137-81 kubenswrapper[7570]: Trace[1633421329]: [1.32702775s] [1.32702775s] END Oct 30 23:57:44.662788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.056260 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.662788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.058423 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.663123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.060603 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.663123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.063149 7570 trace.go:236] Trace[826636244]: "Calculate volume metrics of iptables-alerter-script for pod openshift-network-operator/iptables-alerter-9nwfw" (30-Oct-2026 23:54:09.247) (total time: 56815ms): Oct 30 23:57:44.663123 ip-10-0-137-81 kubenswrapper[7570]: Trace[826636244]: [56.815140635s] [56.815140635s] END Oct 30 23:57:44.663123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.117781 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.663123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.117785 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.663123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.150810 7570 trace.go:236] Trace[372110117]: "Calculate volume metrics of istio-csr-ca-configmap for pod openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" (30-Oct-2026 23:55:05.102) (total time: 1048ms): Oct 30 23:57:44.663123 ip-10-0-137-81 kubenswrapper[7570]: Trace[372110117]: [1.048099553s] [1.048099553s] END Oct 30 23:57:44.663123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.278574 7570 trace.go:236] Trace[1317696843]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" (30-Oct-2026 23:55:05.061) (total time: 1217ms): Oct 30 23:57:44.663495 ip-10-0-137-81 kubenswrapper[7570]: Trace[1317696843]: [1.217502916s] [1.217502916s] END Oct 30 23:57:44.663495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.376341 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:57:44.663495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.474560 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.663495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.573299 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.663495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903727 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.663495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903752 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.663747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903780 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.663747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903762 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.663747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903820 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" podUID="f68436c1-d900-4063-b853-cb9be78ceae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.663747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903819 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.663747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903823 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.663963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903818 7570 patch_prober.go:29] interesting pod/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.663963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903824 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.663963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903880 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.663963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903882 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" podUID="d8235a15-af1b-4332-ac83-1aef5bfb1286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.663963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903785 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.664176 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903848 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.664176 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903923 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664176 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903939 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664176 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.903940 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664176 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.905111 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.664387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.905130 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.664387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.905141 7570 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.664387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.905141 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-t7b8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.664387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.905152 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.664387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.905172 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.905185 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.905175 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.905187 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:06.905172 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664681 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:07.057523 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:07.058615 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:07.061770 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:07.118869 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:07.118942 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.664893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:07.377504 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.665100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:07.474899 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.665100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:07.905951 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.665100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:07.906055 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.665100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:08.006860 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.665100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:08.059673 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": context deadline exceeded" Oct 30 23:57:44.665305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:08.573698 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.665305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:09.979477 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.665305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:10.572832 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.665305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:53:48.563759 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.665305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.323572 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" podUID="b12dc177-11a1-40b0-9fdf-75993ab176e6" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.166:50052: i/o timeout" Oct 30 23:57:44.665609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.333523 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" podUID="7371f1ff-163e-49b2-a4ea-5a9bf3a18215" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.40:50052: i/o timeout" Oct 30 23:57:44.665609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.360666 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.598388968s: [/var/lib/containers/storage/overlay/d42f532082121d22d2dae8c644e5656abc7eae5ca7871bd80e4a5fb632ca6649/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.665609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.360719 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.59846012s: [/var/lib/containers/storage/overlay/be955eb54abbe2b1ab1729ad2779ccc3080ac5870dec9f0f0d8e5f162b4ab12f/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.665609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.360744 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.598483908s: [/var/lib/containers/storage/overlay/2346d8d60b6d752d20206b66a13de33394b7b349ac3cf5d410a2ef3fc2214ff7/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.665609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.360774 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.598512993s: [/var/lib/containers/storage/overlay/a0083eb3d9103d30009b364e78c040c9b52f64998ca8a845bc87ee92cdc39643/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.665609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.360799 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.598535864s: [/var/lib/containers/storage/overlay/494f889ed424fb93575e03fd6dfb34d2d63211134560d5d81e7863454370da4c/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.665846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.360821 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.59855697s: [/var/lib/containers/storage/overlay/4b854bec1dba6649d31a3dff052fa6b8475613425b5d06231467ce5435852f05/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.665846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.360854 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.598584613s: [/var/lib/containers/storage/overlay/6eb16d61180a16d56b421e420725672c9540a33d85e4098b7382a9522d59905c/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.665846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.360881 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.598613085s: [/var/lib/containers/storage/overlay/fa20bee63cf629d4324796d5dc18629b47d2d8c07975c10a454de1437a1e7aa8/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.665846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.360903 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.598633086s: [/var/lib/containers/storage/overlay/474a9d0694eaae7d10a126a3fdc438562c9207c2be0db9868156f76d5e93a76a/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.665846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.360932 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.598656175s: [/var/lib/containers/storage/overlay/b11cc6bca26482597c24c4e75e72edf923855dc8db7f34b3289472a8e3ae1c59/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.665846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.360956 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.597705873s: [/var/lib/containers/storage/overlay/bd64c2e71087c06682d014945c4f9a83c547688a3ba822cda23a6a08d6c484a1/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.666084 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.360983 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.596923172s: [/var/lib/containers/storage/overlay/24975e45960e735faf152b67e4e3d57c07a6c59fa02b9891c80f4dad14df0448/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.666084 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.370989 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" podUID="ca8613ba-3272-43d3-9661-33618f859cff" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.139:50052: i/o timeout" Oct 30 23:57:44.666084 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.377835 7570 trace.go:236] Trace[1925450867]: "Calculate volume metrics of sys for pod openshift-cluster-node-tuning-operator/tuned-qmn89" (30-Oct-2026 23:55:05.650) (total time: 32851ms): Oct 30 23:57:44.666084 ip-10-0-137-81 kubenswrapper[7570]: Trace[1925450867]: [32.85170147s] [32.85170147s] END Oct 30 23:57:44.666084 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:03.412664 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.666084 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.409452 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.666084 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.411248 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" podUID="131867e6-7957-48d7-96d5-50f9e8ff45b2" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.141:50052: i/o timeout" Oct 30 23:57:44.666520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.457875 7570 trace.go:236] Trace[200816545]: "iptables ChainExists" (30-Oct-2026 23:53:49.484) (total time: 109838ms): Oct 30 23:57:44.666520 ip-10-0-137-81 kubenswrapper[7570]: Trace[200816545]: [1m49.838826312s] [1m49.838826312s] END Oct 30 23:57:44.666520 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.457887 7570 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" containerID="57c9e11e3671cbbfd2c731163023febe2b6e3f4d42d12fb2b3c1029eef76c721" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"] Oct 30 23:57:44.666520 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.457897 7570 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" containerID="4b2141d9ec2ac8acb02186b5add60a628ba10bb95e65a20e1a2b415ad3b09bbf" cmd=["crictl","--timeout","10s","info"] Oct 30 23:57:44.666520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.457965 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-7vq6t" podUID="d54e0641-5f0d-41bd-8b1b-900463a24cfa" containerName="extractor" probeResult="failure" output="command timed out: \"crictl --timeout 10s info\" timed out after 10s" Oct 30 23:57:44.666520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.457964 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" podUID="19274393-d63a-4889-8c71-0648e21959a5" containerName="sbdb" probeResult="failure" output="command timed out: \"/bin/bash -c set -xeo pipefail\\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\\novndb-readiness-probe \\\"sb\\\"\\n\" timed out after 5s" Oct 30 23:57:44.666520 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.409440 7570 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" containerID="3de4ff0b7e207fc0a9db19691d7c58c5f4ad48074a8838530d1a6241e96ceca4" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 23:57:44.668668 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.458087 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="da3a65c4-b342-42d7-aad0-29e63b90125a" containerName="prometheus" probeResult="failure" output="command timed out: \"sh -c if [ -x \\\"$(command -v curl)\\\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \\\"$(command -v wget)\\\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi\" timed out after 3s" Oct 30 23:57:44.668668 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.410721 7570 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" containerID="1d36cf57fc43656bb45a7cf70d5ebff3eaf868cded08fc853463cc66b7cc2b3e" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"] Oct 30 23:57:44.668668 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.458122 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" podUID="19274393-d63a-4889-8c71-0648e21959a5" containerName="ovnkube-controller" probeResult="failure" output="command timed out: \"test -f /etc/cni/net.d/10-ovn-kubernetes.conf\" timed out after 1s" Oct 30 23:57:44.668668 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.410710 7570 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" containerID="3de4ff0b7e207fc0a9db19691d7c58c5f4ad48074a8838530d1a6241e96ceca4" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/healthy; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/healthy; else exit 1; fi"] Oct 30 23:57:44.668837 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.458205 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="da3a65c4-b342-42d7-aad0-29e63b90125a" containerName="prometheus" probeResult="failure" output="command timed out: \"sh -c if [ -x \\\"$(command -v curl)\\\" ]; then exec curl --fail http://localhost:9090/-/healthy; elif [ -x \\\"$(command -v wget)\\\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/healthy; else exit 1; fi\" timed out after 3s" Oct 30 23:57:44.668837 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.411769 7570 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" containerID="3a0b7f0fe5f30176b5c2b926930f0e9c3773e1f01096208c6f45acbfc230c6d9" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"] Oct 30 23:57:44.668837 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.458312 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" podUID="19274393-d63a-4889-8c71-0648e21959a5" containerName="nbdb" probeResult="failure" output="command timed out: \"/bin/bash -c set -xeo pipefail\\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\\novndb-readiness-probe \\\"nb\\\"\\n\" timed out after 5s" Oct 30 23:57:44.668837 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.416896 7570 log.go:32] "Status from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Oct 30 23:57:44.668837 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.459776 7570 kubelet.go:3135] "Container runtime sanity check failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Oct 30 23:57:44.668837 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.461346 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": context deadline exceeded" Oct 30 23:57:44.669576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.464312 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" podUID="131867e6-7957-48d7-96d5-50f9e8ff45b2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.141:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.669576 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:39.465608 7570 logging.go:55] [core] [Channel #2838 SubChannel #2843]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.141:50052", ServerName: "10.134.0.141:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.141:50052: operation was canceled" Oct 30 23:57:44.669576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.465654 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 32.94523262s: [/var/lib/containers/storage/overlay/47599597027e0866276d3d51a57e13a1e9329fc2407479b5150ccab4b22820cc/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.669576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.476181 7570 trace.go:236] Trace[1631067899]: "Calculate volume metrics of maas-api-tls for pod odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" (30-Oct-2026 23:55:05.650) (total time: 33825ms): Oct 30 23:57:44.669576 ip-10-0-137-81 kubenswrapper[7570]: Trace[1631067899]: [33.825165201s] [33.825165201s] END Oct 30 23:57:44.669576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.476343 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.669829 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.476433 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.669829 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.477293 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" podUID="6843e367-692b-41f0-a9c6-7704d732f009" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.149:50052: i/o timeout" Oct 30 23:57:44.669829 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.477441 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" podUID="37d7f8cf-f311-4b98-a6b7-25a043ac78ae" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.160:50052: i/o timeout" Oct 30 23:57:44.669829 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.477572 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.669829 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.477598 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.670038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.488273 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-p5xwl" podUID="ee540c2e-f028-44ee-be53-d178b286fb02" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.24:9403/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.670038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.497851 7570 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.670038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.488536 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.670038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.488611 7570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-xsv62 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.670038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.488628 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podUID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.670242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.490108 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.72603478s: [/var/lib/containers/storage/overlay/d00ad0a95a1b422a34c7c8e657719ef28ce71622c44e844bd98583090b8ae134/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.670242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.490421 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" podUID="fb14f0ca-115b-4fe6-afcd-6338df872573" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.153:50052: i/o timeout" Oct 30 23:57:44.670242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.490875 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.670242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.491881 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:57:44.670242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:54:01.317863 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:57:44.670242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.494628 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:57:44.670503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:54:06.142770 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:57:44.670503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.535452 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.670503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:54:06.894410 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 30 23:57:44.670503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.535569 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.670503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.535604 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.670503 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.457874 7570 log.go:32] "ListContainers with filter from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" filter="" Oct 30 23:57:44.670503 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.409443 7570 log.go:32] "ListContainers with filter from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" filter="state:{state:CONTAINER_RUNNING}" Oct 30 23:57:44.670791 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.535662 7570 container_log_manager.go:154] "Failed to rotate container logs" err="failed to list containers: rpc error: code = DeadlineExceeded desc = context deadline exceeded" Oct 30 23:57:44.670791 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.535665 7570 kuberuntime_container.go:554] "ListContainers failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Oct 30 23:57:44.670791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:54:06.928891 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 23:57:44.670791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.535750 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.670791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.535784 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.670791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.535803 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.670791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.535808 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.671070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.535838 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.671070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.535866 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.671070 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.535693 7570 kubelet_pods.go:1275] "Error listing containers" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Oct 30 23:57:44.671070 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.535911 7570 kubelet.go:2707] "Failed cleaning pods" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Oct 30 23:57:44.671070 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.535928 7570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2m48.183s" Oct 30 23:57:44.671070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.535957 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.671070 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.535955 7570 kubelet.go:2545] "Skipping pod synchronization" err="container runtime is down" Oct 30 23:57:44.671070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536022 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jt8kr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.671423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536090 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.671423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536134 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.671423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536143 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7fc6e9-dockercfg-85m2r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.671423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536210 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8bg6s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.671423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536243 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-e15304-dockercfg-8mxkq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.671423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536272 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536283 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536295 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536328 7570 trace.go:236] Trace[1593564868]: "Calculate volume metrics of kube-api-access-5tchs for pod openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" (30-Oct-2026 23:54:07.287) (total time: 92249ms): Oct 30 23:57:44.672354 ip-10-0-137-81 kubenswrapper[7570]: Trace[1593564868]: [1m32.249234749s] [1m32.249234749s] END Oct 30 23:57:44.672354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536328 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-x7g2z\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536380 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536415 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536422 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-v5mxr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672709 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.410897 7570 log.go:32] "ListContainers with filter from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" filter="" Oct 30 23:57:44.672709 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.536483 7570 kuberuntime_container.go:554] "ListContainers failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Oct 30 23:57:44.672709 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.536504 7570 generic.go:254] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Oct 30 23:57:44.672709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536609 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536636 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536668 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536675 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" podUID="7f016eec-7091-4a3e-847e-7a382085ffef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.43:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.672999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536694 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536696 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536741 7570 trace.go:236] Trace[969227491]: "Calculate volume metrics of metrics-certs for pod odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" (30-Oct-2026 23:54:07.332) (total time: 92203ms): Oct 30 23:57:44.672999 ip-10-0-137-81 kubenswrapper[7570]: Trace[969227491]: [1m32.203757639s] [1m32.203757639s] END Oct 30 23:57:44.672999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536720 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kc52k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.672999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536740 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lwvlw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536792 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536814 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9ef03a-openshift-default-dockercfg-mf4wj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536839 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536874 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536885 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dd3a15-openshift-default-dockercfg-72cjz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536905 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-6c5084-openshift-default-dockercfg-bblz7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536950 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536975 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537011 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"distinct-2-w5-dd3a15-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537012 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537049 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537077 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537110 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537149 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7fc6e9-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537180 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537213 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f026c1-openshift-default-dockercfg-6hgm8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537285 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.673783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537294 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537316 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537359 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-djdr2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537365 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537385 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7b0299-openshift-default-dockercfg-cq9kk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537399 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ww7vc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537425 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537451 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"distinct-2-w4-843d8c-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537482 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f0817c\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537504 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537528 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537561 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-82xnp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674498 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:39.537574 7570 logging.go:55] [core] [Channel #2850 SubChannel #2852]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.43:50052", ServerName: "10.134.0.43:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.43:50052: operation was canceled" Oct 30 23:57:44.674498 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.537577 7570 controller.go:251] "Failed to update lease" err="Put \"https://a89e5e1c7169248cead0c4e7f1cca8d1-9b02837b85ac7765.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-137-81.us-east-2.compute.internal?timeout=10s\": http2: client connection lost" Oct 30 23:57:44.674498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.536635 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537627 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674498 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537635 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537659 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537684 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537685 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537707 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-6c5084-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537744 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-lc8kv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537765 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537797 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537814 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537841 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-843d8c\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537852 7570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537584 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2ccd42-openshift-default-dockercfg-qcc6m\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.674973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537906 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"default-dockercfg-4fxkj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537909 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537913 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2nz9z\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537213 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"simulator-w2-9ef03a-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537945 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b567bb-openshift-default-dockercfg-wvjgm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537957 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675476 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537816 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-df3b5f-openshift-default-dockercfg-hmtcd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675476 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537979 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-843d8c-dockercfg-ptlb9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675476 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537990 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-82d449-openshift-default-dockercfg-mhmr7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675476 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537998 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4bbvk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675476 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538018 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537944 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538044 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538073 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zcrv2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538080 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538097 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538129 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ttclb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538143 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538149 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-1j1fi1cun0mic\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538162 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2b4009-openshift-default-dockercfg-mnjpt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538189 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-32732d-dockercfg-rcltc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.675938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538193 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-611ef3\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538200 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-32732d\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538214 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8gxz9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538236 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538252 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538260 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538274 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rb9gl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676420 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538279 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-2q87t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676420 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538303 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-e15304\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676420 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538324 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2mrzn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676420 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538326 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676420 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.537801 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676420 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538338 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538360 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-23f75e\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538364 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538379 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"distinct-2-w0-d3dcca-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538386 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538388 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hc9wr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538402 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"distinct-w4-843d8c-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538419 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538436 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-23f75e-dockercfg-w5fh9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538486 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qmkhv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538275 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a63288\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538500 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-843d8c-openshift-default-dockercfg-l4st5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.676977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538509 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538523 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7fc6e9\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538540 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538549 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hgj94\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538302 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-611ef3-openshift-default-dockercfg-gcsfx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538569 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-82d449\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538576 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-611ef3-dockercfg-m9bvp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538601 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538603 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538229 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k8dwv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538623 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-a3e876-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538632 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"default-dockercfg-ftrjj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538640 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538360 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538654 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-32732d-openshift-default-dockercfg-fvnxc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538659 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d3dcca-dockercfg-vk8p4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538681 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-n7mcg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677707 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538690 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538605 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538712 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538419 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d3dcca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538739 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538742 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.677949 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538738 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-82d449-dockercfg-fwwd9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538767 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d3dcca-openshift-default-dockercfg-mtkk6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538796 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"premium-w0-d3dcca-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538803 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538824 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xmrvg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538849 7570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538855 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a63288-openshift-default-dockercfg-tdk54\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678438 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538872 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678438 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538721 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678438 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538883 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"premium-w3-b567bb-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678438 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.538826 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678438 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.539709 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-df3b5f-dockercfg-rnjbv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.539741 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"default-dockercfg-jtbmr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.539766 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-df3b5f\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.539789 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.540175 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-a3e876-dockercfg-2tm45\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.541792 7570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-xsv62 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.678896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.541821 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podUID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.678896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.542420 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" podUID="d387eda1-c364-4443-ba0d-619368f57f28" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.132:50052: i/o timeout" Oct 30 23:57:44.678896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.542832 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-23f75e-openshift-default-dockercfg-ngphj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.542864 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-a3e876\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.678896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.542926 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"default-dockercfg-75htb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.679124 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.542971 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7b0299\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.679124 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.543011 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.679124 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.543067 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.679124 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.543098 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.679124 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.543067 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" podUID="d387eda1-c364-4443-ba0d-619368f57f28" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.132:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.679374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.543106 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"premium-w4-843d8c-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.679374 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:39.543172 7570 logging.go:55] [core] [Channel #2839 SubChannel #2855]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.132:50052", ServerName: "10.134.0.132:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.132:50052: operation was canceled" Oct 30 23:57:44.679374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.543201 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a63288-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.679374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.544365 7570 trace.go:236] Trace[1151777197]: "Calculate volume metrics of istio-envoy for pod openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" (30-Oct-2026 23:54:33.613) (total time: 65931ms): Oct 30 23:57:44.679374 ip-10-0-137-81 kubenswrapper[7570]: Trace[1151777197]: [1m5.931251512s] [1m5.931251512s] END Oct 30 23:57:44.679374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.548986 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": context deadline exceeded" Oct 30 23:57:44.679374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.549102 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" podUID="a4acfbb3-3ee2-4890-984a-2fc8a796bece" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.158:50052: i/o timeout" Oct 30 23:57:44.679740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.551437 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" podUID="1e8a6ea5-1c91-462f-94fd-7136a48971e9" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.44:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.679740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.552928 7570 patch_prober.go:29] interesting pod/dns-default-h4hmx container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.134.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.679740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.552961 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-h4hmx" podUID="cb76bb5e-c0a6-4a38-8732-40193dd39cac" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.679740 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:39.553571 7570 logging.go:55] [core] [Channel #2857 SubChannel #2858]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.44:50052", ServerName: "10.134.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.44:50052: operation was canceled" Oct 30 23:57:44.679740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.553939 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.679740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.553971 7570 ???:1] "http: TLS handshake error from 10.134.0.14:41366: EOF" Oct 30 23:57:44.679740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.554698 7570 ???:1] "http: TLS handshake error from 10.134.0.14:57824: EOF" Oct 30 23:57:44.680045 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.555715 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.171:50052: i/o timeout" Oct 30 23:57:44.680045 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.556963 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.680045 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.556994 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.680045 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.557024 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:9091/-/healthy\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.680045 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.557055 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/healthy\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.680267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.557159 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.680267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.557180 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.680267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.557228 7570 patch_prober.go:29] interesting pod/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.680267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.557244 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.680267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.557281 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.680555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.557617 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" podUID="e0f6ac75-7dd2-44be-8b54-92ff018b8a3d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.148:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.680555 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:39.559575 7570 logging.go:55] [core] [Channel #2836 SubChannel #2859]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.148:50052", ServerName: "10.134.0.148:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.148:50052: operation was canceled" Oct 30 23:57:44.680555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.572283 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.680555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.572575 7570 trace.go:236] Trace[1045441117]: "Calculate volume metrics of bound-sa-token for pod cert-manager/cert-manager-597b95cb84-p5xwl" (30-Oct-2026 23:55:05.772) (total time: 33800ms): Oct 30 23:57:44.680555 ip-10-0-137-81 kubenswrapper[7570]: Trace[1045441117]: [33.800356063s] [33.800356063s] END Oct 30 23:57:44.680555 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:03.205254 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.680810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.573238 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.680810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.573326 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"default-dockercfg-mdbds\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.680810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.573929 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.680810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.573963 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" podUID="5244915c-9573-489c-b1f8-e22948899df3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.162:50052: i/o timeout" Oct 30 23:57:44.680810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.574550 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.680810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.574723 7570 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mxggs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.681049 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.574758 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" podUID="363febe9-9102-48c9-8faa-09bf49151bf0" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.116:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.681049 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.574791 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.681049 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:03.436954 7570 patch_prober.go:29] interesting pod/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.681049 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.574882 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.681049 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:03.425088 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.681257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.574975 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.681257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.575022 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.681257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.574980 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:57:44.681257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.575079 7570 trace.go:236] Trace[230943764]: "Calculate volume metrics of secret-telemeter-client-kube-rbac-proxy-config for pod openshift-monitoring/telemeter-client-976bbf6cb-lk9st" (30-Oct-2026 23:55:02.449) (total time: 37125ms): Oct 30 23:57:44.681257 ip-10-0-137-81 kubenswrapper[7570]: Trace[230943764]: [37.125974057s] [37.125974057s] END Oct 30 23:57:44.681257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.575198 7570 trace.go:236] Trace[166316928]: "Calculate volume metrics of workload-socket for pod openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" (30-Oct-2026 23:55:01.696) (total time: 37878ms): Oct 30 23:57:44.681257 ip-10-0-137-81 kubenswrapper[7570]: Trace[166316928]: [37.878663721s] [37.878663721s] END Oct 30 23:57:44.681257 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.575514 7570 trace.go:236] Trace[809947938]: "Calculate volume metrics of kube-api-access-jqd2g for pod cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" (30-Oct-2026 23:55:03.307) (total time: 36267ms): Oct 30 23:57:44.681257 ip-10-0-137-81 kubenswrapper[7570]: Trace[809947938]: [36.267645027s] [36.267645027s] END Oct 30 23:57:44.681631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578295 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.681631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578336 7570 trace.go:236] Trace[2115200945]: "Calculate volume metrics of credential-socket for pod openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" (30-Oct-2026 23:55:03.207) (total time: 36371ms): Oct 30 23:57:44.681631 ip-10-0-137-81 kubenswrapper[7570]: Trace[2115200945]: [36.371027413s] [36.371027413s] END Oct 30 23:57:44.681631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578420 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.681631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578529 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.681631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578578 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.681631 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578612 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.681913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578685 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.681913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578728 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.681913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578738 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qwtvf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.681913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578775 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.681913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578781 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n9xrj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.681913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578812 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gskv9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.682154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578828 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rpncv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.682154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578853 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.682154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578869 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-anck8g3riltac\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.682154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578899 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.682154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578904 7570 trace.go:236] Trace[1613951901]: "Calculate volume metrics of openshift-service-ca for pod odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" (30-Oct-2026 23:54:35.763) (total time: 63815ms): Oct 30 23:57:44.682154 ip-10-0-137-81 kubenswrapper[7570]: Trace[1613951901]: [1m3.815790228s] [1m3.815790228s] END Oct 30 23:57:44.682154 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578936 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9tmfn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.682429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578944 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wczpt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.682429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578971 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.682429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.578979 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lzqxs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.682429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579011 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.682429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579020 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.682429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579044 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.683199 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579054 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.683199 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579087 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.683199 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579095 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.683199 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579120 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.683199 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579129 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xj5wh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.683199 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579156 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579163 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579191 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579197 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579227 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579261 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579339 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579357 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z9rpn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579379 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7fh9lroec4t9q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579404 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579431 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579588 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579634 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579658 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-885999-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579670 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579700 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579704 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579741 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"simulator-w0-d3dcca-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.684911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579751 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fl65j\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685158 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579773 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1bc837-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685158 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579788 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-e15304-openshift-default-dockercfg-qbhbd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685158 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579821 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685158 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579852 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"default-dockercfg-rzhfh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685158 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579864 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685365 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579891 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-200408\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685365 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579899 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685365 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579927 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5f557\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685365 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579932 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-200408-dockercfg-zpwrj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685365 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579967 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.579994 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"simulator-w1-611ef3-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.580007 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-86d846-dockercfg-dc664\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.580042 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.580088 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" podUID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:57:44.685602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.580192 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581054 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qbh58\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581094 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581152 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a63288-dockercfg-5m62n\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581189 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1bc837-openshift-default-dockercfg-5htq6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581288 7570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581326 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.685844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581359 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-843d8c-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.686081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581417 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-59scr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.686081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581455 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-885999-dockercfg-tmfpm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.686081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581504 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.686081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581541 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.686081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581602 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.686081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581652 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.686334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581685 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.686334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581741 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.686334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581777 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-885999\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.686334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581811 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2b4009\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.686334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.581885 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-86d846\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.686334 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.582017 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.817201555s: [/var/lib/containers/storage/overlay/985e2ddaac6ff4474c9cca6ab46b56d2e432f8d681b5d8818836978ff483cb1d/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.686600 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.582556 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.697843123s: [/var/lib/containers/storage/overlay/e76fa96356ab2d74ab383725a3dc9fb193c9992a8951b183e6a7512a8d67f527/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.686600 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.583967 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.686600 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.583979 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.686600 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.583997 7570 patch_prober.go:29] interesting pod/metrics-server-5946dc94c7-djbgv container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.686600 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.584023 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" podUID="ed360b42-1a02-4474-9bf1-2117d1b438f1" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.14:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.686810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.583972 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.686810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.584059 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.686810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.584562 7570 patch_prober.go:29] interesting pod/metrics-server-5946dc94c7-djbgv container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.14:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.686810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.584556 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 30 23:57:44.686810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.584585 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" podUID="ed360b42-1a02-4474-9bf1-2117d1b438f1" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.14:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.686810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.584793 7570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeNotReady" Oct 30 23:57:44.687046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.584868 7570 setters.go:546] "Node became not ready" node="ip-10-0-137-81.us-east-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T23:55:39Z","lastTransitionTime":"2026-10-30T23:55:39Z","reason":"KubeletNotReady","message":"container runtime is down"} Oct 30 23:57:44.687046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.585186 7570 trace.go:236] Trace[203052093]: "Calculate volume metrics of model-cache for pod e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" (30-Oct-2026 23:54:42.236) (total time: 57348ms): Oct 30 23:57:44.687046 ip-10-0-137-81 kubenswrapper[7570]: Trace[203052093]: [57.348289503s] [57.348289503s] END Oct 30 23:57:44.687046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:05.783665 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.687046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.585599 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.687046 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.585633 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.687320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.586045 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" podUID="a4acfbb3-3ee2-4890-984a-2fc8a796bece" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.158:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.687320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:54:38.692109 7570 patch_prober.go:29] interesting pod/network-check-target-rhm2r container/network-check-target-container namespace/openshift-network-diagnostics: Readiness probe status=failure output="Get \"http://10.134.0.4:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.687320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.586155 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" containerName="network-check-target-container" probeResult="failure" output="Get \"http://10.134.0.4:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.687320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.586211 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 58.398152123s: [/var/lib/containers/storage/overlay/b343f6190cd826bdbf89b4e18af6964c439aa7c85f197a9bab412ab927385000/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.687320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:54:39.691268 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.687568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.587596 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" podUID="7371f1ff-163e-49b2-a4ea-5a9bf3a18215" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.40:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.687568 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:39.587764 7570 logging.go:55] [core] [Channel #2837 SubChannel #2863]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.158:50052", ServerName: "10.134.0.158:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.158:50052: operation was canceled" Oct 30 23:57:44.687568 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.588596 7570 log.go:32] "ListContainers with filter from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" filter="" Oct 30 23:57:44.687568 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.588630 7570 kuberuntime_container.go:554] "ListContainers failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Oct 30 23:57:44.687568 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:39.588786 7570 logging.go:55] [core] [Channel #2864 SubChannel #2865]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.40:50052", ServerName: "10.134.0.40:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.40:50052: operation was canceled" Oct 30 23:57:44.687568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.595104 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.687818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.606947 7570 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.687818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.606980 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.687818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.607607 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.687818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.607713 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.687818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.607734 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.688029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.608122 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"distinct-2-w6-e15304-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.688029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.608545 7570 trace.go:236] Trace[525687207]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" (30-Oct-2026 23:54:35.864) (total time: 63744ms): Oct 30 23:57:44.688029 ip-10-0-137-81 kubenswrapper[7570]: Trace[525687207]: [1m3.744119101s] [1m3.744119101s] END Oct 30 23:57:44.688029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.608537 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"default-dockercfg-l9wjz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.688029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.608969 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.688029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.609037 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m2.610591804s: [/var/lib/containers/storage/overlay/a5c309176eb202bc4611ad5729e9481e5f5dcdc0bc1555e75b609aef9f1793de/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.688029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.609520 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-9ef03a-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.688321 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.609609 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.688321 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:39.609642 7570 logging.go:55] [core] [Channel #2840 SubChannel #2866]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.45:50052", ServerName: "10.134.0.45:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.45:50052: operation was canceled" Oct 30 23:57:44.688321 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:39.610919 7570 logging.go:55] [core] [Channel #2870 SubChannel #2871]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.171:50052: operation was canceled" Oct 30 23:57:44.688321 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.611636 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" podUID="1d0b4116-23a4-4074-8dfb-93b7bbf53681" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:57:44.688321 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.613295 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.688633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.613331 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.688633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.616295 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" podUID="d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.18:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.688633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.617985 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.688633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.618501 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.688633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:08.058590 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.688847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.619015 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.688847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:08.059666 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.688847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.619542 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.688847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.619797 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.688847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.620132 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.689052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.620551 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.689052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.620866 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.689052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.620931 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.689052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.620953 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.689052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.621030 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.689265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.621052 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.689265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.621109 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-t7b8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.689265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.621165 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.689265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.632437 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.990823687s: [/var/lib/containers/storage/overlay/9a72512fb1e8453ba32c56f101bbfe183acf907cecc90d545d424c7f2798984d/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.632499 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.990899453s: [/var/lib/containers/storage/overlay/c796d0120331a58b0facbe8a1539ed8a61b2d6ee4e6aaec7a1b47586a183302e/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633049 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.991440992s: [/var/lib/containers/storage/overlay/be7b50b699869e83a7ac1d65056123c1d082f7c54a2018e3d1b10581ba23001b/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633200 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.991589562s: [/var/lib/containers/storage/overlay/7f976496ac654b502f09bc10d58caefc03c41018bf2f22f84432e11eae21c528/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633241 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.991631019s: [/var/lib/containers/storage/overlay/3555237851dfd484d63fb219b7272eabf9a650ddd2f1c9719713643e628b8fdc/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633274 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.991661688s: [/var/lib/containers/storage/overlay/77305de2075a06f82b3208c7e7560cd811c938155778626ac4e2c3b65f5ed9a9/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633308 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.991694309s: [/var/lib/containers/storage/overlay/074ed094d175eebb63b56570c16ce9b8eaf7d806112ffb3198173c6cebbbf02b/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633342 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.99172741s: [/var/lib/containers/storage/overlay/3cd3f4dd4bd95bcc4f16248ff6069a0ec287080492705bd254b02c1f982c4ead/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633374 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.991757595s: [/var/lib/containers/storage/overlay/2cc9c21cbee90d0f09cce4f18382e0c5310abcf1d1fbe62dd07fdf1070e5c83a/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633405 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.991787877s: [/var/lib/containers/storage/overlay/d6a83b81c63d8556de56877f75889e61401d2c6ae7943c12dc5bc129a6e45a2c/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633436 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.991817062s: [/var/lib/containers/storage/overlay/06316c3b437d6ded1b6290d6e372fc2272f03e02a4ebba3e7e01f41e79825c7a/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633492 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.991868901s: [/var/lib/containers/storage/overlay/f83d0cf731f9a95cb58ff45611114866f2c39e443f5fbc8f008551999897e67b/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633524 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.991901517s: [/var/lib/containers/storage/overlay/8f7bc021593e4f7d9b01f7e84342af2be71b53546bd6dc866493cb8664ee9ce5/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633556 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.991931728s: [/var/lib/containers/storage/overlay/b4ca2c8d0f6ee43f130900ba13205786ca2befdd26a946c94837243cba5079e9/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.689805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633589 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.991963411s: [/var/lib/containers/storage/overlay/66566d71be382a228158ed6fea7d7e0ef1ddd61f69f2fa6ed7b520ce24769873/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633710 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.992081353s: [/var/lib/containers/storage/overlay/88d1f5a44080969c696f894458fdc76c5e446d178195c5dce4b27b97fdd24b73/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633742 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.992112637s: [/var/lib/containers/storage/overlay/d536dd89d0db5f6c5650dcf9e6f17f9723308eaf3b0892c49837e6d43a78eda1/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633775 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.992142687s: [/var/lib/containers/storage/overlay/ee4c9c9429ca3364553025b204266493e5ce33b0a98897cfbf27c77b2c413e13/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633807 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.992174306s: [/var/lib/containers/storage/overlay/746823caad49564f577a252371db2b4be7b340194a8a33ca6133f45efb7ef022/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633840 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.992204597s: [/var/lib/containers/storage/overlay/a43af1357cc42fa5eb866b9efc92872105da9d3192dc0ffa8e5fe9c1963a9dff/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690043 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633873 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.992237594s: [/var/lib/containers/storage/overlay/7544f7e006bf45a930c0398ed714f843d6ce655a33c58e466a1284b3162d7a06/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690289 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.633907 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m0.992269444s: [/var/lib/containers/storage/overlay/0a7283d12d93303b6c6955a2b817be235c2b58db1d10136a68778413e867f981/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690289 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.637697 7570 kubelet.go:2545] "Skipping pod synchronization" err="container runtime is down" Oct 30 23:57:44.690289 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.637852 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" podUID="a74a48f3-2b68-424b-9489-33e22dab2f2f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.152:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.690289 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.784504 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" podUID="08022df4-0b83-43af-9b14-b85ab9f83be3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:57:44.690289 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.784625 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.099905175s: [/var/lib/containers/storage/overlay/0c80fcd28b531c5e3c1f3f58a56620473b8c8d0e5a197cb86cf5f77f2851f98e/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690289 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.784860 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.100141813s: [/var/lib/containers/storage/overlay/2eeb3bf32e407929b37df7d6fe4af94d541774603eba11d2ccfc64ecab5f460a/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690289 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.785281 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" podUID="ca8613ba-3272-43d3-9661-33618f859cff" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:57:44.690627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.785829 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.101089387s: [/var/lib/containers/storage/overlay/48c3ecbd7827f397cb36f71af58e7ca77d8c67fc3b65ec4810205ee6956c68a1/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.785982 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.101241837s: [/var/lib/containers/storage/overlay/32831e22db5c71996a5a6a89992eeb2442666d85f4470d7dcbaf924fd99550e8/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.787036 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.10227727s: [/var/lib/containers/storage/overlay/bcffdf1dea646997b25bdd8038a1ef13ea9a02a1d9e595ba859f86193375befa/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.787047 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.102282505s: [/var/lib/containers/storage/overlay/81e0195da4be565c10393176e81317418912340a32386428a5e020ff4fc25ab9/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.787166 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.102400073s: [/var/lib/containers/storage/overlay/1d56d4730b83586ee2b9244f407105effe3fe60211ce2278b1aa19d0a55cf1a3/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.787187 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.102424153s: [/var/lib/containers/storage/overlay/b35ce32cae0728601fbf18ad29dd7be995d8bc5de568f813a60404aba4581595/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.787249 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.102480564s: [/var/lib/containers/storage/overlay/a539563af2cda935afdf999700b41dfe741381cf7ec6c01e4d441326dbdc2cfa/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.787368 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.102594499s: [/var/lib/containers/storage/overlay/17d0f665a47fedfd0e68c54683e03c60940fd58ad7aed0d919e4714f3bb274bc/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.787414 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.102642122s: [/var/lib/containers/storage/overlay/dc5af5d3c7e7fcf9c0c70b4189de923f2aa0234e48172879e26c72c8e1b8c014/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.787887 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.659457664s: [/var/lib/containers/storage/overlay/c64ed1e52846196b1aff36847518c9f757300cf6f4193a110cc752c382125ad7/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.787924 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.659496807s: [/var/lib/containers/storage/overlay/914ba0235fd501e4fba33d6df8e2eed9f63e41a4cb9651121d323ee3ae04c2a4/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.788292 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.659861455s: [/var/lib/containers/storage/overlay/2b0c9aa16fb6d0257b94d880d9b9304fce30c9a1bc0787f865071ab22fad4c86/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.788332 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.659901017s: [/var/lib/containers/storage/overlay/c938542bfe36ebe55a7f032ca35fc8973ffd90385f1d9818467057111b4df8a7/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.690907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.788832 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.660396489s: [/var/lib/containers/storage/overlay/7f53709d5e0ec310ca86cf207b17f9b29ee0a395e4142fae97c476917d9a5872/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.788976 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.660522104s: [/var/lib/containers/storage/overlay/669c018af63462489022f768fd3845fd1ea75ec4fa3bdc5917e3927d1fed1d30/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.789073 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.66061701s: [/var/lib/containers/storage/overlay/39b85c1cd6aa5c71333bd1fa13e6cca6931bdfa3befb1f79f32728b1473e8088/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.789163 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.660708572s: [/var/lib/containers/storage/overlay/17564f4b4cace765a15e4fe07b6bfeb5593f062f5630b6b83a4126c0350af905/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.789257 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.660800228s: [/var/lib/containers/storage/overlay/a16a18a38419e4c4ffdbd2a660023655421eeb7bf32ad244f480ec645f1d0b05/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.789260 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.660797702s: [/var/lib/containers/storage/overlay/157ad0624c798eecb9a7a87a4747c63f07eb92ce8a568225a3154f62eba27b90/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.789344 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.660870164s: [/var/lib/containers/storage/overlay/16608acac27528d1128a55aab5ce9caa30776e2e15de879d8f355be8377febaa/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.789428 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.660969565s: [/var/lib/containers/storage/overlay/3cb7bf639476a3490301232391482a3517972ba83734b183c99d68019baa3d22/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.789440 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.660902221s: [/var/lib/containers/storage/overlay/38c009c6bcba31aaed9953484b29942a835e59ff64436b0cbb98875967fa3f78/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.789535 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.661056068s: [/var/lib/containers/storage/overlay/5e5e4ebc24fc6db62f732ae3ea912af2d07e58df14d44ce8bbb21e2899ed0a4e/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.795408 7570 trace.go:236] Trace[1029087746]: "Calculate volume metrics of alertmanager-main-db for pod openshift-monitoring/alertmanager-main-0" (30-Oct-2026 23:54:35.477) (total time: 64318ms): Oct 30 23:57:44.691529 ip-10-0-137-81 kubenswrapper[7570]: Trace[1029087746]: [1m4.318211544s] [1m4.318211544s] END Oct 30 23:57:44.691529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.795734 7570 trace.go:236] Trace[456649089]: "Calculate volume metrics of metrics-client-ca for pod openshift-monitoring/kube-state-metrics-5dc8d7d89b-thls4" (30-Oct-2026 23:54:35.744) (total time: 64051ms): Oct 30 23:57:44.691529 ip-10-0-137-81 kubenswrapper[7570]: Trace[456649089]: [1m4.051065929s] [1m4.051065929s] END Oct 30 23:57:44.691529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.796188 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" podUID="fb14f0ca-115b-4fe6-afcd-6338df872573" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:57:44.691529 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:39.837976 7570 kubelet.go:2545] "Skipping pod synchronization" err="container runtime is down" Oct 30 23:57:44.691529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.845787 7570 trace.go:236] Trace[556696397]: "Calculate volume metrics of whereabouts-flatfile-configmap for pod openshift-multus/multus-additional-cni-plugins-lwrmd" (30-Oct-2026 23:55:06.464) (total time: 33381ms): Oct 30 23:57:44.691529 ip-10-0-137-81 kubenswrapper[7570]: Trace[556696397]: [33.381290438s] [33.381290438s] END Oct 30 23:57:44.691919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.847018 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.641000164s: [/var/lib/containers/storage/overlay/ea7ce0f13bc5b63a1df0df2069ac5614d57372dcc29dcf0f002c6590c6c77ed3/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.847058 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.433198321s: [/var/lib/containers/storage/overlay/e3cbb383bef2480afd1f3dd3687439b3e405e3ee7c54ab8a42adf7d52825e6ff/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.919580 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.496619473s: [/var/lib/containers/storage/overlay/f65febe638815c9c2362f6810c79be9172a3d5c74707f637f630de591b6c59ec/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.920274 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.449095772s: [/var/lib/containers/storage/overlay/cedc84ddfb5da05d6728336f2b94e22d5afa76627fd24632923cce4f1190193b/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.920310 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.449130606s: [/var/lib/containers/storage/overlay/d2d065b7e174e439a27d86e3b269b78b06e80faa352305e8bc9491d4f428d035/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.935223 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.444986733s: [/var/lib/containers/storage/overlay/8a00602211af9a04e5bd2ba50d7fd0099635326780a56175ff118fe764ff42c0/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.691919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.935429 7570 trace.go:236] Trace[154099953]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" (30-Oct-2026 23:55:04.984) (total time: 34950ms): Oct 30 23:57:44.691919 ip-10-0-137-81 kubenswrapper[7570]: Trace[154099953]: [34.950570652s] [34.950570652s] END Oct 30 23:57:44.692254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.941818 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.447944593s: [/var/lib/containers/storage/overlay/3d6dbccaefb2e840fc49eb5118cfccdb9ef635473575b9dadf0c545ae40325f3/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.692254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.945472 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.450972899s: [/var/lib/containers/storage/overlay/4371ea3e29ef70c557520b5998bf6cecd4b8224d1df2a197ed5b493baafe530e/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.692254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.945513 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.449626198s: [/var/lib/containers/storage/overlay/a5aaddb41da4ca8dc2dd02f5f79b7646ab5f42cd545b518de8c38cd2838cba8e/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.692254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.945633 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.44974821s: [/var/lib/containers/storage/overlay/0b16ac04d017400b8610d0c14e3cafc513d682deaa495e2e2936e07419c2f494/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.692254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.945691 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.449802543s: [/var/lib/containers/storage/overlay/e71a74d311e5d41ae9c89e958bfc9fea68c3181e34ea558f1e3f3a5f1f9581ef/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.692254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.945724 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.449835496s: [/var/lib/containers/storage/overlay/c636f2a6b869fe32f6486fbc51f4a95a0f2cee760ad2de3ac00318b2976403ab/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.692254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.945823 7570 trace.go:236] Trace[1916688488]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" (30-Oct-2026 23:54:35.766) (total time: 64179ms): Oct 30 23:57:44.692254 ip-10-0-137-81 kubenswrapper[7570]: Trace[1916688488]: [1m4.179369771s] [1m4.179369771s] END Oct 30 23:57:44.692737 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.946020 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.450127081s: [/var/lib/containers/storage/overlay/d67a7a519d1eb9e0dafc605f4c77d7b76211e23491dd9eb3f3da2fd62fc9a886/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.692737 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.946054 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.450159709s: [/var/lib/containers/storage/overlay/86f7573597615e31798c4913aea40b423e47d36340bc185a20bdb3b2104bb701/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.692737 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.946141 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.450243816s: [/var/lib/containers/storage/overlay/9fdde1d43c8e886219ef7e56ff0a3a77f5443de2df398630faae9abccf308ea3/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.692737 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.946173 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.450274455s: [/var/lib/containers/storage/overlay/e3430d604f15f0169162f2ffb8bfa0afcfc048aa1b1b06f7bd486ecec5b14670/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.692737 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.946185 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.450285068s: [/var/lib/containers/storage/overlay/8935841c6148e4b7a531c6c320e7152a7c4db82cc3e4f9bf49f8998fc50864e1/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.692737 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.946201 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.450299672s: [/var/lib/containers/storage/overlay/f459b0fbbbcc7d649dde1a6a5116c7d0210f508da5719170911912539f8642af/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.692737 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.946217 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.450313515s: [/var/lib/containers/storage/overlay/1202b29307516df9f75bc2ca4ac0aac0c292081d929857db1c4c5fe24e7c829b/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693019 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.946231 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.450326564s: [/var/lib/containers/storage/overlay/5e3e9ee239e6552347cd310a996cb5cdcbd4ce4664d11c3076a698957c5b86ba/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693019 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.946260 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.450346836s: [/var/lib/containers/storage/overlay/e0bf4953faad6bdda91e07902849651ca8882e3f47bff9c71ab46dae0d16d3aa/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693019 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.946262 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.4503443s: [/var/lib/containers/storage/overlay/6807975fa144519fe3e5caa7e2445247a8c0bee201fa5c55d746ef0eb156c01b/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693019 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:39.946321 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.450402071s: [/var/lib/containers/storage/overlay/42f322701453299d0b29576a3d8ec035fc7d358ccf6e2c01b862fe5e36013562/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693019 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.006996 7570 trace.go:236] Trace[1484591307]: "Calculate volume metrics of config-volume for pod openshift-dns/dns-default-h4hmx" (30-Oct-2026 23:54:35.473) (total time: 64533ms): Oct 30 23:57:44.693019 ip-10-0-137-81 kubenswrapper[7570]: Trace[1484591307]: [1m4.533609475s] [1m4.533609475s] END Oct 30 23:57:44.693019 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.053222 7570 trace.go:236] Trace[951498466]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" (30-Oct-2026 23:55:03.492) (total time: 36560ms): Oct 30 23:57:44.693019 ip-10-0-137-81 kubenswrapper[7570]: Trace[951498466]: [36.5604812s] [36.5604812s] END Oct 30 23:57:44.693019 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.056710 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.604825161s: [/var/lib/containers/storage/overlay/957d622a6fabd4799224e4d53b8f3e9f9ea2fd462402970a970726a6f22ed99c/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693381 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.102627 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.470868067s: [/var/lib/containers/storage/overlay/4c950915cc723ba039102dfbad0cda02d23df8d660859e2c1c06093e65fa85cf/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693381 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.108805 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.475646739s: [/var/lib/containers/storage/overlay/5c97f7fca644f155cbc88e963c09ba16f7c4992ebbc48dc815a0de121e59a3e7/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693381 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.108838 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.417005271s: [/var/lib/containers/storage/overlay/30a95659f789fecb08f3df7cf332705eed7af73dfb8015737aada935505bb2be/diff /var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-g86k2_9c20fb11-2dcf-445e-96cd-4d97045a2ab9/monitoring-plugin/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693381 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.108864 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.377634693s: [/var/lib/containers/storage/overlay/90de72811eee09b24f55d1f736deab02ddefc2cdb75aec9c5f869a2700cbdf71/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-t49gb_68aefc1f-0a32-4722-a494-054f46f44b5b/kube-rbac-proxy-self/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693381 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.108883 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.377654014s: [/var/lib/containers/storage/overlay/13fc7ee8866eb5ddedaf4630865a3c9a1891b4ed40a9701fd90e47e992e1bb86/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/northd/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.108901 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.377670789s: [/var/lib/containers/storage/overlay/9f79876b5c8da341e967f5ae2fd88177f82cf63931bbb7a089621385bbd7160b/diff /var/log/pods/openshift-monitoring_node-exporter-hq9sq_a0371e4c-ff4b-4867-98a1-bb598ab03845/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.110061 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.37882577s: [/var/lib/containers/storage/overlay/35271418e1c37de4b7888549ea8717976558ef1cb86ef91daf881eb4bb08bde8/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.110087 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.378855406s: [/var/lib/containers/storage/overlay/0bd751342d0b910bc952cc203f08c3bb411454f4fe1deb6fc808e804fda4ade0/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.110111 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.470812597s: [/var/lib/containers/storage/overlay/08611c58bbdf5b145b7040b5ee6ae228b796baee8fb21f34f82f8c7f0f67db58/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.110141 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.470840022s: [/var/lib/containers/storage/overlay/760db5d6d5047157f14b5e8449ce74c3b64d5cf1319dc89316361b40f769ce8b/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.110172 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.470868849s: [/var/lib/containers/storage/overlay/da822d9d0d2cdc047728088fb162965d14bd57f89e054ac489a8a0951619325f/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.110617 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.471278665s: [/var/lib/containers/storage/overlay/ed69edde03bd3cf330de915fdf2d1745aaefec3d19f696e00325a04acf427fb9/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.110640 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.379407931s: [/var/lib/containers/storage/overlay/aead9544bd40cc159d1e6a7d220f9f1cabe7f12e5883de5c718c7299e4a254c0/diff /var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/prom-label-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.110665 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.471327474s: [/var/lib/containers/storage/overlay/f970cf9b131c498179e307ab4dd658e12f12a11f611fd57307929351e16f8e90/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.110895 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.471550621s: [/var/lib/containers/storage/overlay/dab79fcd87c2164205e19ce63791e7c29135e8859d59cf53cc174bba34c780c9/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.693928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.110923 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.471577182s: [/var/lib/containers/storage/overlay/d1e79e55d07e98318d43a08e65fccda9981d13408266608ba28173bf3bbfa0ce/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.110958 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.471610879s: [/var/lib/containers/storage/overlay/1a92d06cf4bfceb45c1bc8b6dcf4f705f3a8784bf4e25d66a55bafc2f9693ea1/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.110982 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.471633266s: [/var/lib/containers/storage/overlay/df497565b00750a47fe246b6f31a1075e9fa09432a7cdad68b73411ceaf09f99/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.111232 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.471878582s: [/var/lib/containers/storage/overlay/b4092aa5d2eeecf888ef63e1d8f29c78545760cf40bffd8cf15c8bb74a0decd5/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.111265 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.471910749s: [/var/lib/containers/storage/overlay/f236b901b18be822152c6c8bebafe441af5461e6105dd29741cafa752b838e0b/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.111302 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.471945977s: [/var/lib/containers/storage/overlay/1ef75a8ed5ab38a35a72a5768007686c92ae81418e40b004b7b856ab5d301165/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.111378 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.472014446s: [/var/lib/containers/storage/overlay/4a9ca6492ee7e04fab87480d95a5815cc97346f8f60905725b9d73b15e613a24/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.111433 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.472064793s: [/var/lib/containers/storage/overlay/258bf7784b694d591e85c7b4fc11e027661540fae0d21917105b06a115db1442/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.111454 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.380219751s: [/var/lib/containers/storage/overlay/8f57ae5b6c03f7a5ffbfd5484509cc17d28ce1246c6ade4d693c20b243a67b6a/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.111538 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.380300607s: [/var/lib/containers/storage/overlay/8fcf25bc01cedf84ba899e5c3eb6261f2177d1fa7eb08356011187d35a0f19a9/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/alertmanager/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.115072 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.32783765s: [/var/lib/containers/storage/overlay/fb3b23827a51a6b2b216726749359d819a5f30c31024eb7644c047f0cc6fa845/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.178801 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m49.42052532s: [/var/lib/containers/storage/overlay/30b4a6775958a4ef1a2b395cf67610d8b6c311908f801be25df2cebe10e12d71/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5_e8c5ef3a-fb2d-4f45-9d16-470f553b62c5/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.178828 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.053215976s: [/var/lib/containers/storage/overlay/b84a05b52289da70061eec39f056a06299c2328aff3d97d572eba8b93ab97836/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.178861 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m47.516206235s: [/var/lib/containers/storage/overlay/7e97f3d8d436ca6beb296709e5aca670ebcfb0dd4aabd2b68f7b863b769db52b/diff /var/log/pods/openshift-ingress_e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk_51164b20-de0a-4b55-bdb7-7c25c2d5d922/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.178891 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 34.608502892s: [/var/lib/containers/storage/overlay/378ef04e65b7c86c840fd91c2cf1ea7b4a7c407e830d266802a402fc354353e4/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.178917 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 34.608528145s: [/var/lib/containers/storage/overlay/c87476079796d311df4f2d7bb830a092f2d5c1600b87b655e1602a496fc9f961/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694919 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:40.189791 7570 controller.go:251] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"ip-10-0-137-81.us-east-2.compute.internal\": the object has been modified; please apply your changes to the latest version and try again" Oct 30 23:57:44.694919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196813 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 34.607509867s: [/var/lib/containers/storage/overlay/268a50d29b6e6e8466bff3ec7625c48e04fb9255017b862b1e14731166cec6f4/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.694919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196848 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.465080283s: [/var/lib/containers/storage/overlay/1b779171f801079be98b1c067503ccbdf021c24d100122b866226e7349db6559/diff /var/log/pods/cert-manager_cert-manager-597b95cb84-p5xwl_ee540c2e-f028-44ee-be53-d178b286fb02/cert-manager-controller/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196870 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.959584879s: [/var/lib/containers/storage/overlay/5dfb9d812d54804edb00ba844f1a5386ca8ec555bcb79c209fbded98159a4438/diff /var/log/pods/openshift-image-registry_node-ca-vqvzf_f5f12d67-0b46-434f-8c41-a230d421eefd/node-ca/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196883 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.279634393s: [/var/lib/containers/storage/overlay/88cb8fa866137edd068803e272efab84dc735ed9ba27c9971d12ae783f3c7189/diff /var/log/pods/openshift-ingress_e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz_48390cfe-87a3-4b4e-9986-8adec60f6349/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196873 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.465101908s: [/var/lib/containers/storage/overlay/fd98498c4d8b710748c4ae2ca27074d51aa81467be4cb8008b2567f3234a8e94/diff /var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5n89j_20fc3e5f-362e-41f7-bb52-2b171d7af818/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695193 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196893 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.452107135s: [/var/lib/containers/storage/overlay/bba6625d7bd500d93dc1ed92064604daff63f485fb138e04e8c58084b08fb286/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2_131867e6-7957-48d7-96d5-50f9e8ff45b2/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196901 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.465133762s: [/var/lib/containers/storage/overlay/3e9c1e5d01d16b92e7db38570c8265c37322278731bdbc94c37ce0c007779620/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g_aa08b804-5246-4fb8-a68c-cf7af9ba4a54/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.695401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196929 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.074149836s: [/var/lib/containers/storage/overlay/08419f2c0de795fb01ba6d4390a44dd91bed83ec4efd02d3c8095df295384160/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196935 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.911545686s: [/var/lib/containers/storage/overlay/7987722b0a09024dd25f7742bfd3a70f70b03b81c2489afc401532b530d1bb24/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196945 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.437563035s: [/var/lib/containers/storage/overlay/7ea5f4fa2a20fc73e73740f5ddd51ebd7b63eb122f5f18031f79f5619b8a64cb/diff /var/log/pods/openshift-ingress_e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69_84a7842f-6043-4b93-ab32-d25e2ba8c5bb/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196960 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.434707373s: [/var/lib/containers/storage/overlay/7cc5070b57b476e4399ea865b3647b685fc83ae06e290144316c4114b9ab2ad3/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-node-driver-registrar/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196965 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.434712092s: [/var/lib/containers/storage/overlay/f2c679e7cd5f2bfc6b2642f521367d05b8cdda5b25d215a32900f18b69b6ece9/diff /var/log/pods/openshift-multus_network-metrics-daemon-jgq5s_8cad31bd-cac3-44ca-bb2c-ca302c4e82e4/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.196983 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.434726273s: [/var/lib/containers/storage/overlay/5a5349eee9b0bc6f4fa9ea36707cac98493aa281628d3808950374b0d31b6a9a/diff /var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/kube-rbac-proxy-rules/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197004 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.478645489s: [/var/lib/containers/storage/overlay/16e961f567e71eedb110a42b6f947d298dc907a9c18619a15074bb0879ad3f37/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2_b12dc177-11a1-40b0-9fdf-75993ab176e6/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695615 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197008 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.432944728s: [/var/lib/containers/storage/overlay/daa585bcebbabe4fec4b3941b1eaed6ce79b204a87bc255f0d0c36f7c6397f9b/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-p5kqz_9a41f90f-265b-4a47-a2bc-9239bbd1822b/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695774 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197018 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.508679443s: [/var/lib/containers/storage/overlay/96eefe0d9887665678dd4d2c31afd891226e59681f332d9aefcab1d5c58a06e5/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm_1e8a6ea5-1c91-462f-94fd-7136a48971e9/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695774 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197029 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.474347959s: [/var/lib/containers/storage/overlay/e5883ba613520bba70ea60e98be6000acd1a7c947b0f509e4ea4e80af26bcf77/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp_522aa71b-47e4-44b1-825c-bf81305d21b4/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695774 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197033 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.999718455s: [/var/lib/containers/storage/overlay/2571647ccfb8952151e56eaff4f7dff6e45bc09d764392a89769ca755476a238/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg_37d7f8cf-f311-4b98-a6b7-25a043ac78ae/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695774 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197047 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.716105639s: [/var/lib/containers/storage/overlay/1d6721638d637bce25faa19702cfca7d58a751db555957aa13c0046620be55b4/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll_a74a48f3-2b68-424b-9489-33e22dab2f2f/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197058 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.999779875s: [/var/lib/containers/storage/overlay/44b0704d11c7dc38b323f41e81949db45c4189a4af6346a34d0c7c7250d7f222/diff /var/log/pods/e2e-models-e2e-worker-w3-b567bb_premium-w3-b567bb-kserve-586f4c9b9-rn44f_ed4e6002-31dc-4eb3-9988-d808eee76c7f/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197064 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.432999661s: [/var/lib/containers/storage/overlay/34a96be9ca0d791025cb1b421a2ac627992ca521fb66ed7f31b0143677ead44b/diff /var/log/pods/openshift-ingress_kube-auth-proxy-dfd47b86c-gjwxh_2b7be075-e8bb-4db1-bfab-782276c29b6e/kube-auth-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197079 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.719335617s: [/var/lib/containers/storage/overlay/a4d268a8a8717d1a62b159ce196714489eee4b826ca22119f35fb77eda7664ca/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb_d387eda1-c364-4443-ba0d-619368f57f28/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.695946 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197080 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.728898484s: [/var/lib/containers/storage/overlay/d48b62b69845920830050d5e6fd4dfcef959e40ce97b979c3b7bfd3b7240198d/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp_e0f6ac75-7dd2-44be-8b54-92ff018b8a3d/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197093 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m58.923110882s: [/var/lib/containers/storage/overlay/12fbd2a1baf816c08f27674a73b2c44b1b8d54a47b3b85cab8394c7d94dfa7a7/diff /var/log/pods/e2e-models-e2e-worker-w0-d3dcca_premium-w0-d3dcca-kserve-64955fd666-hx4zl_3e57b4d8-7b73-4490-9138-de9401d6fd65/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197103 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.739516505s: [/var/lib/containers/storage/overlay/5953d0ad13461ca6560c0d4709522e12222222fd97e9131178fdf34eee1fb1f1/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx_6843e367-692b-41f0-a9c6-7704d732f009/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197110 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.302934486s: [/var/lib/containers/storage/overlay/eb6ba043451ed0c3e71aa76c7105b75f5e1db582169d62dc2beefb4fcb9b96e1/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg_7f016eec-7091-4a3e-847e-7a382085ffef/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197114 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.434865573s: [/var/lib/containers/storage/overlay/eb10cfe189a72077416d9595382c191d23f6a1ec9520e0c5ebf60082667efb79/diff /var/log/pods/openshift-dns_dns-default-h4hmx_cb76bb5e-c0a6-4a38-8732-40193dd39cac/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696323 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197122 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.47413626s: [/var/lib/containers/storage/overlay/c9e0b4b340c24fa81a582eac02eb31743c12ef9f25efbd3e606ccfab05aed162/diff /var/log/pods/openshift-ingress_e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24_b94094c6-876a-4082-ad7d-f35bb1909f87/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696323 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197127 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.434882088s: [/var/lib/containers/storage/overlay/1fd92a325a73f706efe6ce154a5764ec96d70f811c06fd3b10df36b85aaaa079/diff /var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal_70da0cbd0028ec04eeaec823ad5f3973/haproxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696323 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197136 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.198697607s: [/var/lib/containers/storage/overlay/32f4ef24912ac2375218657860eb7c8f91cc7af57d07641a0404d19548193274/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/prom-label-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696323 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197143 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.43489238s: [/var/lib/containers/storage/overlay/485d48f9d9ab722e64040886487d1f3dbdac88493c58ca2ee5902e8e6bb03f3c/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-thls4_3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/kube-state-metrics/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197146 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.433075779s: [/var/lib/containers/storage/overlay/ceb920e1e7a3e793f85b86c40fbd5f83c50e42cd965ae018e76950fe864d5206/diff /var/log/pods/openshift-ingress_e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b_cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197157 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.555556359s: [/var/lib/containers/storage/overlay/39e54a0677170c09896d6f03393ffd52de086d46bda748378b344b5506f2b596/diff /var/log/pods/openshift-monitoring_telemeter-client-976bbf6cb-lk9st_d0b4de16-5f57-44fe-a273-a77ebd48d050/telemeter-client/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197158 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.555564558s: [/var/lib/containers/storage/overlay/60fd7647cad01e28ecb0b6fa0b7fb4214c64cceae69cc10da083d8561c386c38/diff /var/log/pods/openshift-monitoring_telemeter-client-976bbf6cb-lk9st_d0b4de16-5f57-44fe-a273-a77ebd48d050/reload/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197168 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.555568704s: [/var/lib/containers/storage/overlay/5ced9e81568b36ea8a7b027b5a76151c2ec151920cd1f56c22ec859ffd7ca5e4/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696702 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197171 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.555568918s: [/var/lib/containers/storage/overlay/7b8f5dbc0f03541bfbadfe7770e8cf48e4ee208a73aed4b4a3db480d7aa99a1b/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-lwrmd_312d3272-40b0-41c2-8131-bafd6266a11e/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696702 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197178 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.555574063s: [/var/lib/containers/storage/overlay/7dae0ecfe0d8187b7dbfbb65052d8195244515a6a820bd37827468962332fc52/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696702 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197181 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.555554334s: [/var/lib/containers/storage/overlay/47793c876c4c5d01e87c8506c573d65ce5cc0467d2ea9e7fd3d9e89277e28eb8/diff /var/log/pods/openshift-network-diagnostics_network-check-target-rhm2r_a10c4ed2-0361-47e1-8810-8a710b2be42d/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696702 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197191 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.512490746s: [/var/lib/containers/storage/overlay/e68cb84172be4a39333cd7abcdfcd538da59c3e7519e017364f0f7c93ea51c65/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696866 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197194 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.555586792s: [/var/lib/containers/storage/overlay/0dac6c286fb0b5b6447b29ec1f51f2c6c01d9332b32717150983cfb854a43d02/diff /var/log/pods/openshift-insights_insights-runtime-extractor-7vq6t_d54e0641-5f0d-41bd-8b1b-900463a24cfa/extractor/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696866 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197203 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2m1.463784013s: [/var/lib/containers/storage/overlay/2b215fe76accc9bae0d0931ffd6d952c738e9fb935e12cf2401502156062262d/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb_bfa36aaf-a967-456c-a544-9a2bcc4ac7d3/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696866 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197208 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.512487407s: [/var/lib/containers/storage/overlay/e402921a279de129be7427fee7b99527e8a0e21029c9e1a5f9fbb74cfcd1e91d/diff /var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/thanos-query/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.696866 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197215 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.512487579s: [/var/lib/containers/storage/overlay/b830dad46c70a107f568121442e17d6c57ed98e9fedb48bfcbe803575f44c6b8/diff /var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-137-81.us-east-2.compute.internal_f383e62f25895fcb5217aa3ca64dca1a/kube-rbac-proxy-crio/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197222 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.107459867s: [/var/lib/containers/storage/overlay/52376acc02e0f243a72a19182304da6dbecad3d68ff1fade52ae6da04a76ffc0/diff /var/log/pods/openshift-ingress_e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4_e8fab750-eae1-4f10-b8f3-3db73b00a1f9/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197224 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.512494464s: [/var/lib/containers/storage/overlay/53a2f0ab3acd08f98c4309cd58184f7de269803337be73f113d8fdd5ad4472f1/diff /var/log/pods/openshift-monitoring_metrics-server-5946dc94c7-djbgv_ed360b42-1a02-4474-9bf1-2117d1b438f1/metrics-server/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197235 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.512490053s: [/var/lib/containers/storage/overlay/9a9f4be3a49018715bcb4d1c4d5cb65b74468a2e4e310a505a73f5b8e1792030/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197244 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.512501305s: [/var/lib/containers/storage/overlay/90efbca770046be08c5d002f48b022752d474c92671e020681e23e6db5e7cd98/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/config-reloader/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197268 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.51253408s: [/var/lib/containers/storage/overlay/4d86043d70c001d12673c73c993554093cf74312485ab5672b56588b1fc45be0/diff /var/log/pods/openshift-multus_multus-nkhth_89205e33-cb00-47fe-b78d-cc1a05d1f123/kube-multus/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197254 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.51250597s: [/var/lib/containers/storage/overlay/9be523497e0fe151cfc66edcd237c2e48a0d02540d5fba80d676ec78482569c5/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/kube-rbac-proxy-metric/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197287 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.512533031s: [/var/lib/containers/storage/overlay/815810e51a6e87afbade1c836ec9b1910d94fe93ec5545786f97c29233ab3971/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/nbdb/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197300 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.512548074s: [/var/lib/containers/storage/overlay/75f0108cf238c31c0bcd6dfa87947c79b29edd38a4eeee18c9d56aa443c060c0/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/kube-rbac-proxy-thanos/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197306 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.068884939s: [/var/lib/containers/storage/overlay/beb2779a43a009c6356e35f50542f5321cac999f5dae351ed0adea127279521f/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/prometheus/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197313 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.068885489s: [/var/lib/containers/storage/overlay/a644fda2555618d5eec515891b49f1c6f690b61a09095fd4f7c8337e3aff41cd/diff /var/log/pods/kube-system_konnectivity-agent-89shx_d3eeb220-e1ba-46e2-94c8-2bb19e02dde9/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197324 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.068896738s: [/var/lib/containers/storage/overlay/620ee42d53049268fd47d0f1a39664ca22d1fdff365003b2296756adf18bac33/diff /var/log/pods/openshift-insights_insights-runtime-extractor-7vq6t_d54e0641-5f0d-41bd-8b1b-900463a24cfa/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197329 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.068899038s: [/var/lib/containers/storage/overlay/a404bf6c87b5c53e3878ddb51210498f4557b3e883c2ccd89e7b100ce5a1b5b9/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-thls4_3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/kube-rbac-proxy-self/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197329 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.512586181s: [/var/lib/containers/storage/overlay/56baec63f55948cea80d82961f0c5560a1f7f4c57db35b0f4a7c40841e4af8b8/diff /var/log/pods/openshift-monitoring_node-exporter-hq9sq_a0371e4c-ff4b-4867-98a1-bb598ab03845/node-exporter/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197340 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.068892122s: [/var/lib/containers/storage/overlay/a255baffe46fcba829c6110007ac4ed6c37edae172e754e6991d3d93804591bb/diff /var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/kube-rbac-proxy-metrics/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197346 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.433273952s: [/var/lib/containers/storage/overlay/9f4d56ce97ccbddd884bdaf82ad15694b5ea0de030e70e61bc178b003829a92f/diff /var/log/pods/openshift-ingress_e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp_61c25f85-fbde-4c00-94c9-502637e57ba5/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197362 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.433295921s: [/var/lib/containers/storage/overlay/178e54113f69810cf2114c33a77d5d377253a1a1b94133160924ae9ae8a2f352/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl_a4acfbb3-3ee2-4890-984a-2fc8a796bece/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197361 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.068928981s: [/var/lib/containers/storage/overlay/cf678d272fc7976b06627168ba82c61f095fb38590fbddc535232b9343eb059f/diff /var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697822 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197387 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.43331699s: [/var/lib/containers/storage/overlay/c85c869fda9cc50e4cdcac383e3458aacd25870ed3ae9ea9ada931827e9e3928/diff /var/log/pods/openshift-ingress_e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9_29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697822 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197390 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.068938176s: [/var/lib/containers/storage/overlay/d551dc3cbadeb52c8a5b049bedf918811efef59a638d472aa5ecacd24d78b586/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697822 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197349 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.068900116s: [/var/lib/containers/storage/overlay/668e0e3acf19878b8218741e211ca94eaadbea2cc637389ebcfb51a52e223217/diff /var/log/pods/openshift-multus_network-metrics-daemon-jgq5s_8cad31bd-cac3-44ca-bb2c-ca302c4e82e4/network-metrics-daemon/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.697822 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197400 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.267814819s: [/var/lib/containers/storage/overlay/6b150c5edd7b1ca69f5e1b404aa3bcd052e95d83ed98d6bc55e1f5abd1c1bb35/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z_c15181ee-3e9a-40c9-a029-9ee504a971fa/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698000 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197409 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.433335845s: [/var/lib/containers/storage/overlay/dc96c17fd475929835eebbdb4acc0176ce21564a081f59b7dfbe09fc27de2e46/diff /var/log/pods/e2e-models-e2e-worker-w4-843d8c_distinct-2-w4-843d8c-kserve-8644666f64-vhwnn_d073cc7c-6856-4131-8bf7-250400f05d7f/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698000 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197415 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.117291891s: [/var/lib/containers/storage/overlay/d3e41270e106e2e9755a87e8c5fb22722dc048a67ed8221a5648c22d3984b3b1/diff /var/log/pods/e2e-models-e2e-worker-w6-e15304_distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl_ee6148fe-e516-492b-b5a7-755a75e82c58/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698000 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197415 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.43334008s: [/var/lib/containers/storage/overlay/c98a22f1fd04912f9efa076a547dd3bd39426a4efe5407ae848bcad8b8f23e08/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght_b4e6db40-bf7c-4440-bed0-8ec6e70cab0f/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.698000 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197429 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.068886483s: [/var/lib/containers/storage/overlay/f973cad01573d376a84630445560caa82c3138f0a6ba74fb9d19bd0b3f3d1fdc/diff /var/log/pods/openshift-ingress_e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798_c9e446a0-c6da-40ab-aa1f-f3756e39619a/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197431 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.308927984s: [/var/lib/containers/storage/overlay/6be4816b08509c82430a0d49d48da911e6724b5bacbaa526b682fea5602b8f39/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4_ca8613ba-3272-43d3-9661-33618f859cff/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197447 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.726277297s: [/var/lib/containers/storage/overlay/ba41d637e03969830c21052341bcc41c1619da647d62e7beb452718a39bd1e32/diff /var/log/pods/openshift-insights_insights-runtime-extractor-7vq6t_d54e0641-5f0d-41bd-8b1b-900463a24cfa/exporter/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197451 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.267866125s: [/var/lib/containers/storage/overlay/cd3058ba6d2cf52b3aeb9de37169925252ba5677fc23be231ae11bff86ef11a3/diff /var/log/pods/e2e-models-e2e-worker-w4-843d8c_premium-w4-843d8c-kserve-7844d66896-27fxd_4f1bda07-310d-43a8-a926-9004487b0034/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.698540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197455 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.77547097s: [/var/lib/containers/storage/overlay/1108fec57902ce5126303aeb6c6007e464c3a285cfeb7121f254f723b05f1df8/diff /var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-x6pgq_2b47b3cc-9735-45f1-8d87-7edd6ef95dd8/jobset-operator/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197524 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.726351633s: [/var/lib/containers/storage/overlay/ccf9cc9a8f0788e3f609bf98fb0a6f3bd51c7847ffa988af8d88613a99b8d9ba/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/thanos-sidecar/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197534 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.726361498s: [/var/lib/containers/storage/overlay/a9bc14739a0a3e7422083cb29e10413933c8853bc30cabdd63e83445b4b6c032/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/ovn-controller/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197538 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.726364553s: [/var/lib/containers/storage/overlay/0c09db27888916a85d91c941bc52228061c76fa94b736a8a3dc7e8996ff1a7af/diff /var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197547 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.433484162s: [/var/lib/containers/storage/overlay/3dcc54b8479d324b621d5d8aef12882854d539507de35502e7c8c3f5c9dfb9e7/diff /var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-vnmqd_baaa353f-43a5-43e5-a73c-92d1840ebdae/cert-manager-webhook/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698915 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197558 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.069012938s: [/var/lib/containers/storage/overlay/ec332e0f46b7ba7d9795eb4bdc5cc199e37edddb3407efb9995e04836440258a/diff /var/log/pods/opendatahub_odh-model-controller-769875db56-4blbf_1793bf9b-92db-4dae-bfd6-f86e5b076739/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698915 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197557 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.069018236s: [/var/lib/containers/storage/overlay/4b3faf0029ea6a1ccb158e79150f8ca77b731ff0151cd603c0ad850fe13179f9/diff /var/log/pods/openshift-ingress_e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf_f179a0a0-c464-4a8b-b6cc-04a21610a832/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.698915 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197567 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.779002311s: [/var/lib/containers/storage/overlay/a7aa06682d5ba76031c6af7007cd8eaac1c0af689bcdd8877a5331a56724a289/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5_c070c544-43c5-4796-b385-6c4094a63624/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.698915 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197573 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.701690353s: [/var/lib/containers/storage/overlay/5faf94b0bcee2a001c75fad5757ab50e1cbe0c0c704256136fb2dec70ee3ea9a/diff /var/log/pods/openshift-network-operator_iptables-alerter-9nwfw_97edeeaf-64b2-4628-a669-5fc121c8ee2f/iptables-alerter/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197582 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.701695581s: [/var/lib/containers/storage/overlay/c4dcae9a167c26250c827408ebc57ee2fa7496cbfcb1dcf9e64c5f3afad71885/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197593 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.701700644s: [/var/lib/containers/storage/overlay/a7f81b84613e83a3bf86bdc99730a8ac65af3bb823b46c216c917feb8d203c79/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/config-reloader/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197604 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.701707908s: [/var/lib/containers/storage/overlay/48554265d83ff926268710f46b984028e52680fd6021993e647cc43f95b565f4/diff /var/log/pods/openshift-dns_node-resolver-bwf5h_1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197611 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.701720304s: [/var/lib/containers/storage/overlay/204242ae1cde4dd33bdfafc786b22ed5c635f3c1229b2e586d1533433e1a1de8/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-t49gb_68aefc1f-0a32-4722-a494-054f46f44b5b/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699285 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197615 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.772574818s: [/var/lib/containers/storage/overlay/8f54da0a78596d861902bd46c804f7cad6acf955f67af06c73b41663d2c38df0/diff /var/log/pods/e2e-models-e2e-worker-w4-843d8c_distinct-w4-843d8c-kserve-df87d4cff-tz6fp_80dc7b17-04ea-4d03-a843-24891d3cda07/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699285 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197624 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.693850583s: [/var/lib/containers/storage/overlay/4bcfc2e73e803edc6350b7e0219f16061b0b0154c38ef68fc9b30848eef9d488/diff /var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-68r7m_347b9a5b-729f-419a-bce5-749ac4aca5b5/cert-manager-cainjector/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699285 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197630 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.313090545s: [/var/lib/containers/storage/overlay/2d62442eabbf1ded93ad20b55998de2371ea1396cc7ac9f43a1a9bb40b0f8c27/diff /var/log/pods/openshift-ingress_e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx_f68436c1-d900-4063-b853-cb9be78ceae6/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699285 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197639 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.303464799s: [/var/lib/containers/storage/overlay/e09987724efa3cf21524ab84afb430b55533805bb956a307ab47256cb255761c/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mxggs_363febe9-9102-48c9-8faa-09bf49151bf0/discovery/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197644 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.069102477s: [/var/lib/containers/storage/overlay/0a161f603ddfa64eb35c23099fb0a2b72d085fea8d70ad02d1375d6d034bb28c/diff /var/log/pods/e2e-models-e2e-worker-w1-611ef3_simulator-w1-611ef3-kserve-8455bfc968-b9llt_00b04bdb-908f-4ced-b3d2-1627d96513b6/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.699577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197651 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.703782581s: [/var/lib/containers/storage/overlay/9f074815449dfeb2c6adcd6fdde2b1517ee82a0a4d04d335711556b364004ffb/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc_12efb180-e881-40a3-9001-8a39a2ddd352/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.699577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197661 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.069119259s: [/var/lib/containers/storage/overlay/f3fdf2a3fa513daa7796a64ce8075659af08ad472684f89a30e46ae75d82d011/diff /var/log/pods/openshift-ingress_e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l_cf77418c-72fd-40ad-b389-df9273f36cfc/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197681 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.693901446s: [/var/lib/containers/storage/overlay/f0631351a60de912d395b3e14337dbdacc4b4f496b2f42a3a01d46349b9e8909/diff /var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2_5275c1e3-85ea-449f-b526-7c3cbb14ed4c/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699777 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197686 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.693904352s: [/var/lib/containers/storage/overlay/53894e6b146e5677d226eef0d3d36ba72db36ebdfa61a0477624ed839c3c58dd/diff /var/log/pods/openshift-ingress_e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl_a6f473a6-9517-4367-b935-294dbd28ceec/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.699777 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197699 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.773931201s: [/var/lib/containers/storage/overlay/a914f565ef768a0a346be25ac688af1bc707b2b0345a7fc6c3969bd8bb3d2dc2/diff /var/log/pods/openshift-ingress_e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx_570b6b11-715d-4a67-9360-78aa45580d57/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699777 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197707 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.703836028s: [/var/lib/containers/storage/overlay/e91783390aa97d397922076705ae546742cf327cb4412039da1bc9f0d0be7b8e/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws_5244915c-9573-489c-b1f8-e22948899df3/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.699777 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197720 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.568108571s: [/var/lib/containers/storage/overlay/01bc050ff3ede245fc5b6578e08988263fdbbae208afaf2849addc5a2ddefe4b/diff /var/log/pods/openshift-ingress_e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj_4c7f3b9a-978c-49f2-a52a-5293dfb1924f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.699993 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197726 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.069186751s: [/var/lib/containers/storage/overlay/6a8fc60645b598451596d1e8407ac9bf942b9229630a08205b8bbe6683d5e429/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc_6b3d259c-4f6c-4033-b6dd-1059616fe3b4/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.699993 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197738 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.693962918s: [/var/lib/containers/storage/overlay/f0723e62218d24ac3caf102b2bef1d64af6ea4758cdc5b0f62128eef666a2440/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq_201fbcb8-8fed-44e3-b60d-4b89d8c114ca/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699993 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197746 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.438366691s: [/var/lib/containers/storage/overlay/5b90dd098e4d643f1d5a31069cb1bf326cf53938c3f35db3115ff7673a58c617/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8_fb14f0ca-115b-4fe6-afcd-6338df872573/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.699993 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197760 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.558456004s: [/var/lib/containers/storage/overlay/d56d6140bc7d2e5c56253a9225cca73da7b31f755ae2d850dc9134491ab1586b/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-thls4_3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197777 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.558469309s: [/var/lib/containers/storage/overlay/05fd72b3c91f09e3e27d0f3dcbc88b11565ca12103c50c8ba3883064f99b1120/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-t49gb_68aefc1f-0a32-4722-a494-054f46f44b5b/openshift-state-metrics/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197785 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.437476444s: [/var/lib/containers/storage/overlay/a28a107902b3427100e50707b355494f8e1097b6f186562f4cb05c85c51bb041/diff /var/log/pods/openshift-ingress_e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk_af49f007-5f21-43cc-b4f3-d78e31b45100/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197797 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.55848917s: [/var/lib/containers/storage/overlay/f8e3539a2bfe6da4be25bcb30af5428ab1c1d3d6fab81f0b22035429b5c673ee/diff /var/log/pods/openshift-dns_dns-default-h4hmx_cb76bb5e-c0a6-4a38-8732-40193dd39cac/dns/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197806 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.558496637s: [/var/lib/containers/storage/overlay/a7c0e7c8693fc8a257065b652fd5066c36bccef876a0e04c0ee9f8a197bc99a5/diff /var/log/pods/openshift-ingress-canary_ingress-canary-9tcb2_284dba08-fe12-4942-a5fd-6c1e2f979a17/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197817 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.558474153s: [/var/lib/containers/storage/overlay/26ac99fb02a37e4926c08c8653a404ff86c95c4d78e127e481ea02248e842af9/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-qmn89_88de885b-31c5-4ce7-a66f-0a212087affd/tuned/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197825 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.558473649s: [/var/lib/containers/storage/overlay/a49f35122a945905681209db5ea0ab4a8b935a5cb82ae723cc1ff4dd04653201/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/sbdb/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197836 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.568228643s: [/var/lib/containers/storage/overlay/b3a6db58fb09b4d0c2798278ae2643b265542d44375feee3ad946e1ec68a7f42/diff /var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-6pxqs_f87cf2ab-c80c-4680-9c10-497e2a11235b/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197839 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.43753111s: [/var/lib/containers/storage/overlay/fa06a709a51529569c198f78feb00dfa6f88a021a5da5689e1edbd6007e984a0/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx_cc2a6d9c-ec3e-438a-9d6b-db75fa30a903/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197845 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.069302032s: [/var/lib/containers/storage/overlay/792adc4a646bbf2e22f785ca95dead4522d7f98381f9398e19b12c532d7adc97/diff /var/log/pods/openshift-ingress_e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd_a74b8dd0-7067-44fd-94c0-948568ca285c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197858 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.15832082s: [/var/lib/containers/storage/overlay/ac5eaa0944ce2c0dad54c06c9c14404c9a7d378cb21855d1933155f8353a8355/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6986c789f7-5bhpc_d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197861 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.566104076s: [/var/lib/containers/storage/overlay/e593b8801075c6881923384cd19c2b28622b2fac411e5f27d5644bb6a7ac66d7/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx_08022df4-0b83-43af-9b14-b85ab9f83be3/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197870 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.567071441s: [/var/lib/containers/storage/overlay/d91c0104ace8b482e2dad547d0f33bacdc7842fbd1c6fb1ccd958fb5a4d16f83/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-7rp6c_e7406dbc-b7ad-4b0c-ac37-c85b302bf76e/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197884 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.410651014s: [/var/lib/containers/storage/overlay/e6f8c59141778017279d386690b2eac72a1855b721e05047a44da311020b5367/diff /var/log/pods/openshift-ingress_e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6_e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197884 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.694096788s: [/var/lib/containers/storage/overlay/3850d1bc8fbc7eb22abcc4a003e89aeb5ab4318b239a2b288cd6c99729884848/diff /var/log/pods/openshift-ingress_e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd_cbf9885e-6fe7-4c33-b91a-60b11bf4b730/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.700788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197891 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m6.584322087s: [/var/lib/containers/storage/overlay/3aa1b53568922c8825d9dde751c035f12b5eb8ac8bf6695fac5a7451543db9ec/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c_e9c9e5ed-908c-463d-8f4f-395a80ababdd/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700788 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197902 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.694115384s: [/var/lib/containers/storage/overlay/ebf739cfecd58d11e198de589f2102d9749ef735c638b4e295e1e748954c54c9/diff /var/log/pods/openshift-ingress_e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq_67262faa-4e17-4e80-9d44-917ef37e70db/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197906 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m13.717367386s: [/var/lib/containers/storage/overlay/9149486066c386ec10e44139fc13f1e35814bfa8aaf1c609db95aec5b45cc446/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt_1d0b4116-23a4-4074-8dfb-93b7bbf53681/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197912 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m14.92493606s: [/var/lib/containers/storage/overlay/435030370c99fd90348ec3c51bf775f6d3b9c47c677f9838ab182a8edc20e383/diff /var/log/pods/openshift-ingress_e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk_d8235a15-af1b-4332-ac83-1aef5bfb1286/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197922 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.559065814s: [/var/lib/containers/storage/overlay/c08014ec14fb4421165ffa63f45e8f2812548ee47468644b0418f79bc2b0f6ad/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn_2533ebe5-857f-4255-8c85-5a526fe1062b/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.700968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197927 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.410690558s: [/var/lib/containers/storage/overlay/63747dd0209d2941b1b813e2a90b60760a4d2fd53d2e0f2bee93209ea8add28b/diff /var/log/pods/e2e-models-e2e-worker-w0-d3dcca_simulator-w0-d3dcca-kserve-74cd68d695-9s4b4_08cd1e28-c399-4cba-951e-0210a978109f/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.701231 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197930 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.694152238s: [/var/lib/containers/storage/overlay/a516443d85784522e54f82a9309c805882c7a29c08ef516c769fec340092039a/diff /var/log/pods/e2e-models-e2e-worker-w2-9ef03a_simulator-w2-9ef03a-kserve-cdb8584b5-76gjs_223c27af-605a-42a8-a29c-ae1fa2b2ace6/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.701231 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197941 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.568331731s: [/var/lib/containers/storage/overlay/69a1b67a43a502a6a2d76337b00d6bf126a091f49f8f5595e1cb19843da575ce/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-t7b8g_02d621d9-9a83-45c7-bef8-9255f543c6f0/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.701231 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197943 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 34.239395661s: [/var/lib/containers/storage/overlay/9640cee11f897f54ce412f9f51a65f38b489af493e772d7cc178bb778d3dc805/diff /var/log/pods/kube-system_global-pull-secret-syncer-6sfch_6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.701231 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.197955 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 43.131854375s: [/var/lib/containers/storage/overlay/873cbd9d8b2d3f4a4daf1a22daaec3ffb0c03312db0d4c0683ebb7aca7696e7a/diff /var/log/pods/openshift-monitoring_telemeter-client-976bbf6cb-lk9st_d0b4de16-5f57-44fe-a273-a77ebd48d050/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.701424 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.219538 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.43079492s: [/var/lib/containers/storage/overlay/1f44acf446b79c4eeef279d5c019e358a07c00b124e437316480cb88324a473c/diff /var/log/pods/openshift-ingress_e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724_d2658215-b85d-44d6-8d22-9803dd36e9f4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.701424 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.220216 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.434356734s: [/var/lib/containers/storage/overlay/8549b996ed862bedf8bbe0d0e6d867083d48dff353f38ee03b336ac9bf5c52b6/diff /var/log/pods/openshift-ingress_e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr_1017ca7a-a16e-44bd-af55-d8babaa94e01/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.701424 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.225031 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.437790465s: [/var/lib/containers/storage/overlay/ca3f279b0d073526d1fe7e0960f66b4e79def3c3cdd34e0da948566960bdf993/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh_eedd4bdb-1495-47e6-884d-5ce26a05e72b/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.701424 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.237181 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m12.235952415s: [/var/lib/containers/storage/overlay/fe49837001233bfe83175e53f6914495bbf08c293f12e080bada70b26eadec99/diff /var/log/pods/e2e-models-e2e-worker-w0-d3dcca_distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb_42fdc936-a982-454f-a538-0a6494d1cfa2/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.701424 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:40.238769 7570 kubelet.go:2545] "Skipping pod synchronization" err="container runtime is down" Oct 30 23:57:44.701674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.250059 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.611409348s: [/var/lib/containers/storage/overlay/ce21d6da650dbbfd9753673e28424d581b97e8ff8622c073b0da6189d3609be4/diff /var/log/pods/openshift-ingress_e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7_31585445-5877-4947-8df5-f30317948127/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.701674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.259224 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.755437007s: [/var/lib/containers/storage/overlay/5208d4cca466afc2340fecbbdbdaf54e0318e89daf9479af1370e69dacb7f283/diff /var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-c6888_eeffad66-ea6b-486f-abd8-63e68120af4c/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.701674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.263535 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.4762957s: [/var/lib/containers/storage/overlay/92f05b9f61b382bef562b8b2aa19f2058a54394f67b3d84e5c39d6fcc307ab60/diff /var/log/pods/openshift-ingress_e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp_f3856a78-100e-48e0-aa1e-9a04bb10c99d/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.701674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.276818 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.107388896s: [/var/lib/containers/storage/overlay/c4f93fe15e5a9ad51f8b72d54bff1b29bbbb4a273ba52fb02b2d86cfb90e177a/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt_7371f1ff-163e-49b2-a4ea-5a9bf3a18215/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.702487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.331599 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 34.761204854s: [/var/lib/containers/storage/overlay/bcba7dcacbd58646714997736c6e7a16279f4eee41283f3e635085ad29b7bfff/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5_06e6c870-08ba-4b3c-a26d-45397d5ca730/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.702487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.344164 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-df3b5f-dockercfg-rnjbv\"" Oct 30 23:57:44.702487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.351657 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-23f75e\"" Oct 30 23:57:44.702487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.357257 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-a3e876\"" Oct 30 23:57:44.702487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.363910 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.702487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.366136 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k8dwv\"" Oct 30 23:57:44.702487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.375825 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"distinct-2-w0-d3dcca-kserve-self-signed-certs\"" Oct 30 23:57:44.702487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.380448 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:57:44.702487 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.395253 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:57:44.702873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.395685 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.6084414s: [/var/lib/containers/storage/overlay/f39dcaf820ff00166e8c80e107d5fdb36b312bc3fe68e119f0a22871a807ba49/diff /var/log/pods/kuadrant-system_authorino-74dc7574c6-6rws8_3dc85af3-2ef5-4005-a8c5-d515a92d2eee/authorino/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.702873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.397087 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 23:57:44.702873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.397964 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:57:44.702873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.408668 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ww7vc\"" Oct 30 23:57:44.702873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.412157 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a63288\"" Oct 30 23:57:44.702873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.414353 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.702873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.414562 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-86d846-dockercfg-dc664\"" Oct 30 23:57:44.702873 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.415564 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.703208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.417217 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.913432261s: [/var/lib/containers/storage/overlay/70fdb3a403ae61abbfa9d076234bdabd603c276cad5bf4aef6ae75ef402d0cfe/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb_4af4419c-fffd-463e-9577-94bad299abc6/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.703208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.417668 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.946486926s: [/var/lib/containers/storage/overlay/cc8a1b4516cc9e7213e654a25f68a4ce41b17570a17435d3cbfdaf9510aa01f0/diff /var/log/pods/e2e-models-e2e-worker-w5-dd3a15_distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r_809bef14-259b-4a21-b170-a6a701e00e6c/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.703208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.419098 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.703208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.419715 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 23:57:44.703208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.436839 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4bbvk\"" Oct 30 23:57:44.703208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.436979 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-843d8c-serving-cert\"" Oct 30 23:57:44.703208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.437128 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v5mxr\"" Oct 30 23:57:44.703208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.437204 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.703208 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.446767 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 23:57:44.703657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.446848 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:57:44.703657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.457983 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.703657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.458067 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.703657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.458838 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.703657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.458889 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.703657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.461858 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.958074736s: [/var/lib/containers/storage/overlay/c28f4e8a2c51f59ca420e282df53a9df60396e35d9f7da10932335d0f3b91e34/diff /var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx_dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.703657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.463818 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gskv9\"" Oct 30 23:57:44.703657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.463893 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.704004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.463970 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 23:57:44.704004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.464059 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xmrvg\"" Oct 30 23:57:44.704004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.464142 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2b4009-openshift-default-dockercfg-mnjpt\"" Oct 30 23:57:44.704004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.464203 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:57:44.704004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.464271 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-86d846\"" Oct 30 23:57:44.704004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.464415 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" podUID="201fbcb8-8fed-44e3-b60d-4b89d8c114ca" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.138:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.704004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.474832 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5f557\"" Oct 30 23:57:44.704004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.474927 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8gxz9\"" Oct 30 23:57:44.704004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.474995 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.704388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.481839 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m12.742304587s: [/var/lib/containers/storage/overlay/1d382d86ece2071185bb89cc0a01ab5774ba0076942f4a8c4e654c71c3ce26c3/diff /var/log/pods/openshift-ingress_e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv_4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.704388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.483913 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 23:57:44.704388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.495713 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-23f75e-dockercfg-w5fh9\"" Oct 30 23:57:44.704388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.495952 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.704388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.500312 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 23:57:44.704388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.500422 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.704388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.500567 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 23:57:44.704388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.500650 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xj5wh\"" Oct 30 23:57:44.708247 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.500729 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.708247 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.501106 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" podUID="c070c544-43c5-4796-b385-6c4094a63624" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.42:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.708247 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.502767 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.708247 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.503238 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.708247 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.503577 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.708247 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.504684 7570 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-g86k2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.708736 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.504737 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" podUID="9c20fb11-2dcf-445e-96cd-4d97045a2ab9" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.15:9443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.708736 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.505610 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a63288-dockercfg-5m62n\"" Oct 30 23:57:44.708736 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.507060 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.708736 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.507098 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.708736 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.507843 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:57:44.708736 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.508108 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.708994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.508934 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 23:57:44.708994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.509407 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-611ef3-openshift-default-dockercfg-gcsfx\"" Oct 30 23:57:44.708994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.509545 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" podUID="37d7f8cf-f311-4b98-a6b7-25a043ac78ae" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.160:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.708994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.510629 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.708994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.514734 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-2q87t\"" Oct 30 23:57:44.708994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.514827 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-df3b5f-openshift-default-dockercfg-hmtcd\"" Oct 30 23:57:44.708994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.514880 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.708994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.514926 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 23:57:44.708994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.514990 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.709367 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:40.515090 7570 logging.go:55] [core] [Channel #2841 SubChannel #2842]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.138:50052", ServerName: "10.134.0.138:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:57936->10.134.0.138:50052: use of closed network connection" Oct 30 23:57:44.709367 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.521754 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.709367 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.521796 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.709367 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.522127 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7b0299-openshift-default-dockercfg-cq9kk\"" Oct 30 23:57:44.709367 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.533996 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 23:57:44.709367 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.534110 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2nz9z\"" Oct 30 23:57:44.709367 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.539044 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d3dcca\"" Oct 30 23:57:44.709367 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.539151 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fl65j\"" Oct 30 23:57:44.709710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.539296 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"distinct-2-w5-dd3a15-kserve-self-signed-certs\"" Oct 30 23:57:44.709710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.539698 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.709710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.539793 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dd3a15-openshift-default-dockercfg-72cjz\"" Oct 30 23:57:44.709710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.539825 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-x7g2z\"" Oct 30 23:57:44.709710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.539901 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.709710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.540489 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.709710 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:40.542659 7570 logging.go:55] [core] [Channel #2845 SubChannel #2846]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.42:50052", ServerName: "10.134.0.42:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:37464->10.134.0.42:50052: use of closed network connection" Oct 30 23:57:44.709990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.542831 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.709990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.542934 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" podUID="6843e367-692b-41f0-a9c6-7704d732f009" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.149:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.709990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.545572 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.709990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.545609 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.709990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.550924 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2b4009\"" Oct 30 23:57:44.709990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.551004 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-32732d\"" Oct 30 23:57:44.709990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.551025 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 23:57:44.710330 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:40.551663 7570 logging.go:55] [core] [Channel #2848 SubChannel #2849]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.160:50052", ServerName: "10.134.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:53396->10.134.0.160:50052: use of closed network connection" Oct 30 23:57:44.710330 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.554606 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded" Oct 30 23:57:44.710330 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.554683 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.710330 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.554862 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 23:57:44.710330 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.554952 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.710330 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.555553 7570 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.710716 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.555610 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.710716 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.574719 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.710716 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.574809 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:9091/-/healthy\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.710716 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.574829 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/healthy\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.710716 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.574937 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" podUID="b12dc177-11a1-40b0-9fdf-75993ab176e6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.166:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.710942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.575074 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.710942 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:40.584733 7570 logging.go:55] [core] [Channel #2853 SubChannel #2854]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.149:50052", ServerName: "10.134.0.149:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:35210->10.134.0.149:50052: use of closed network connection" Oct 30 23:57:44.710942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.585101 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 23:57:44.710942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.585175 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.710942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.585327 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"premium-w4-843d8c-kserve-self-signed-certs\"" Oct 30 23:57:44.710942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.585396 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.710942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.585548 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 23:57:44.710942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.585641 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jt8kr\"" Oct 30 23:57:44.711311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.585705 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.711311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.586540 7570 patch_prober.go:29] interesting pod/metrics-server-5946dc94c7-djbgv container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.711311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.586623 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" podUID="ed360b42-1a02-4474-9bf1-2117d1b438f1" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.711311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.586845 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.711311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.586957 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.711648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.587063 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.711648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.587488 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.711648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.587538 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.711648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.587575 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.711648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.587701 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 46.073344254s: [/var/lib/containers/storage/overlay/e3665a62bfc0dfd6fa6ce6260d72c4e844ec3f4b9ca453eee6cd2c040b5d6087/diff /var/log/pods/openshift-ingress_e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28_ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.711876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.588041 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.711876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.588058 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.711876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.588072 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.711876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.589008 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.711876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.589402 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.711876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.589508 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:57:44.711876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.597615 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.712171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.597672 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.712171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.602751 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 23:57:44.712171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.602866 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hc9wr\"" Oct 30 23:57:44.712171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.606724 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-200408-dockercfg-zpwrj\"" Oct 30 23:57:44.712171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.606791 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 23:57:44.712171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.610723 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:57:44.712171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.610877 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1bc837-serving-cert\"" Oct 30 23:57:44.712171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.610931 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.712535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.611229 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.712535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.613028 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.712535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.613614 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"default-dockercfg-mdbds\"" Oct 30 23:57:44.712535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.615668 7570 patch_prober.go:29] interesting pod/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.712535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.615715 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.712535 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.616102 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wczpt\"" Oct 30 23:57:44.712783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.616542 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.712783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.616582 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.712783 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:40.616732 7570 logging.go:55] [core] [Channel #2860 SubChannel #2861]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.166:50052", ServerName: "10.134.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:56244->10.134.0.166:50052: use of closed network connection" Oct 30 23:57:44.712783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.618698 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.712783 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.618751 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.713006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.619551 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:57:44.713006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.620270 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.713006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.620329 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:57:44.713006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.621529 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.713006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.621564 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.713217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.621609 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded" Oct 30 23:57:44.713217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.621643 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.713217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.621892 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:57:44.713217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.622165 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 23:57:44.713217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.624045 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.713425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.624076 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.713425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.624385 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/odh-model-controller-769875db56-4blbf" podUID="1793bf9b-92db-4dae-bfd6-f86e5b076739" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.713425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.630234 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7b0299\"" Oct 30 23:57:44.713425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.630450 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.713425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.630592 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.713425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.630658 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.713425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.633407 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-a3e876-serving-cert\"" Oct 30 23:57:44.713747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.644596 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" podUID="a74a48f3-2b68-424b-9489-33e22dab2f2f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.152:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.713747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.648722 7570 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.713747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.648774 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.713747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.649004 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.713747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.649079 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.713747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.649148 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-885999\"" Oct 30 23:57:44.713747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.649198 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.713747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.649260 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 23:57:44.714092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.649332 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-e15304-openshift-default-dockercfg-qbhbd\"" Oct 30 23:57:44.714092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.653631 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.714092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.653674 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.714092 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:40.653730 7570 logging.go:55] [core] [Channel #2872 SubChannel #2873]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:41336->10.134.0.171:50052: use of closed network connection" Oct 30 23:57:44.714092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.653858 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.714092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.654216 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.714353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.654252 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.714353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.654655 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.714353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.654688 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.714353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.656701 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 23:57:44.714353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.656731 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9ef03a-openshift-default-dockercfg-mf4wj\"" Oct 30 23:57:44.714353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.656753 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.714353 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:40.661575 7570 logging.go:55] [core] [Channel #2867 SubChannel #2868]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.45:50052", ServerName: "10.134.0.45:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:39830->10.134.0.45:50052: use of closed network connection" Oct 30 23:57:44.714675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.661724 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.714675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.661748 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.714675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.662703 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:57:44.714675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.662757 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-885999-serving-cert\"" Oct 30 23:57:44.714675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.666696 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 23:57:44.714675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.667595 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ttclb\"" Oct 30 23:57:44.714675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.667678 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-885999-dockercfg-tmfpm\"" Oct 30 23:57:44.714675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.668990 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.714675 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:40.671348 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.715034 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.142984 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.715034 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.143006 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" podUID="fb14f0ca-115b-4fe6-afcd-6338df872573" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.153:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.715034 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.143188 7570 trace.go:236] Trace[1459293416]: "Calculate volume metrics of metrics-client-ca for pod openshift-monitoring/telemeter-client-976bbf6cb-lk9st" (30-Oct-2026 23:55:39.945) (total time: 2197ms): Oct 30 23:57:44.715034 ip-10-0-137-81 kubenswrapper[7570]: Trace[1459293416]: [2.197933191s] [2.197933191s] END Oct 30 23:57:44.715034 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.756512 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2m2.872905724s: [/var/lib/containers/storage/overlay/9c08cbdbd940b063d50924e2f8b7c36951b33066cffbbb96219f325d0aed3c22/diff /var/log/pods/openshift-ingress_e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr_f8b86c0c-3f2f-4f9d-be26-948cb9084373/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.715034 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.143208 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.715034 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.756695 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-611ef3-dockercfg-m9bvp\"" Oct 30 23:57:44.715351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.143224 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.715351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.143376 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:57:44.715351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.756940 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"default-dockercfg-4fxkj\"" Oct 30 23:57:44.715351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.143392 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:57:44.715351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.143418 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:57:44.715351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.143433 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.715628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.143457 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:57:44.715628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.143728 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 23:57:44.715628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.143851 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-82d449\"" Oct 30 23:57:44.715628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.144501 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:57:44.715628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.144809 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 23:57:44.715628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.144836 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 30 23:57:44.715628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.145090 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:57:44.715937 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.145104 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:57:44.715937 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.759073 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:57:44.715937 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.145140 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.715937 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.759144 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.715937 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.145157 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.715937 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.759209 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.716197 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.145363 7570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:57:44.716197 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.759760 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.716197 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.148726 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:57:44.716197 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.148745 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Oct 30 23:57:44.716197 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.148760 7570 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-g86k2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9443/health\": context deadline exceeded" start-of-body= Oct 30 23:57:44.716197 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.759917 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" podUID="9c20fb11-2dcf-445e-96cd-4d97045a2ab9" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.15:9443/health\": context deadline exceeded" Oct 30 23:57:44.716197 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.149073 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.716574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.759989 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.716574 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:42.149967 7570 kubelet.go:2545] "Skipping pod synchronization" err="container runtime is down" Oct 30 23:57:44.716574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.760399 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kc52k\"" Oct 30 23:57:44.716574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.150275 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.716574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.150286 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:57:44.716574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.156640 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.716831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.760733 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.716831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.185626 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.716831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.761482 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.716831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.191587 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.716831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.761558 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.717051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.197590 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/odh-model-controller-769875db56-4blbf" podUID="1793bf9b-92db-4dae-bfd6-f86e5b076739" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.717051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.530596 7570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-xsv62 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.717051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.762077 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podUID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.717051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.575799 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.717051 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.582593 7570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-xsv62 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.717270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.762337 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podUID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.717270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.621050 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.717270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.765065 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.717270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.765128 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-611ef3\"" Oct 30 23:57:44.717270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.766169 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:57:44.717270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.766804 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 23:57:44.717270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.766947 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f0817c\"" Oct 30 23:57:44.717270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.767682 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 23:57:44.717270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.767813 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.717270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.768328 7570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:57:44.717733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.769103 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:57:44.717733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.770369 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-djdr2\"" Oct 30 23:57:44.717733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.770520 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"distinct-2-w6-e15304-kserve-self-signed-certs\"" Oct 30 23:57:44.717733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.771887 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.717733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.772009 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 23:57:44.717733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.772183 7570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:57:44.717733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.772334 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-9ef03a-serving-cert\"" Oct 30 23:57:44.717733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.773208 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-843d8c-openshift-default-dockercfg-l4st5\"" Oct 30 23:57:44.717733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.773235 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.717733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.773311 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hgj94\"" Oct 30 23:57:44.717733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.774735 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"default-dockercfg-ftrjj\"" Oct 30 23:57:44.717733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.774854 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"distinct-w4-843d8c-kserve-self-signed-certs\"" Oct 30 23:57:44.718190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.774867 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8bg6s\"" Oct 30 23:57:44.718190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.776435 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7fc6e9\"" Oct 30 23:57:44.718190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.776449 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-6c5084-serving-cert\"" Oct 30 23:57:44.718190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.776565 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-843d8c\"" Oct 30 23:57:44.718190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.776568 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2ccd42-openshift-default-dockercfg-qcc6m\"" Oct 30 23:57:44.718190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.776594 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-82d449-dockercfg-fwwd9\"" Oct 30 23:57:44.718190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.777986 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 23:57:44.718190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.778011 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.718190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.778014 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.718190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.778090 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.718190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.778127 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.718190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.778159 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 23:57:44.718727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.779476 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.718727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.779521 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-n7mcg\"" Oct 30 23:57:44.718727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.779542 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"simulator-w1-611ef3-kserve-self-signed-certs\"" Oct 30 23:57:44.718727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.779584 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 23:57:44.718727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.779606 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"default-dockercfg-rzhfh\"" Oct 30 23:57:44.718727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.781096 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 23:57:44.718727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.781125 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b567bb-openshift-default-dockercfg-wvjgm\"" Oct 30 23:57:44.718727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.781176 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 23:57:44.718727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.781248 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 23:57:44.718727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.781262 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 23:57:44.718727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.781251 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.718727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.781337 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 23:57:44.719210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.781443 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-32732d-openshift-default-dockercfg-fvnxc\"" Oct 30 23:57:44.719210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.782136 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:57:44.719210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.782230 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qbh58\"" Oct 30 23:57:44.719210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.782303 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 23:57:44.719210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.782624 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.719210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.782652 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 23:57:44.719210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.782720 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"distinct-2-w4-843d8c-kserve-self-signed-certs\"" Oct 30 23:57:44.719210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.782741 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:57:44.719210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.784198 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.719210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.784307 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d3dcca-dockercfg-vk8p4\"" Oct 30 23:57:44.719210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.785824 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.719210 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.785844 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 23:57:44.719759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.785909 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.719759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.785922 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-e15304-dockercfg-8mxkq\"" Oct 30 23:57:44.719759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.787565 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 23:57:44.719759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.787588 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-df3b5f\"" Oct 30 23:57:44.719759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.787589 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 23:57:44.719759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.787623 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"premium-w3-b567bb-kserve-self-signed-certs\"" Oct 30 23:57:44.719759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.787645 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 23:57:44.719759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.787625 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 23:57:44.719759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.787660 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:57:44.719759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.787669 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 23:57:44.719759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.788911 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 23:57:44.719759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.788920 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.720258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.788930 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.720258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.788973 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 23:57:44.720258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.788981 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d3dcca-openshift-default-dockercfg-mtkk6\"" Oct 30 23:57:44.720258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.788994 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 23:57:44.720258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.789038 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"premium-w0-d3dcca-kserve-self-signed-certs\"" Oct 30 23:57:44.720258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.790525 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 23:57:44.720258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.790536 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-200408\"" Oct 30 23:57:44.720258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.790587 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.720258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.790655 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a63288-openshift-default-dockercfg-tdk54\"" Oct 30 23:57:44.720258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.790668 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.720258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.791691 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-6c5084-openshift-default-dockercfg-bblz7\"" Oct 30 23:57:44.720258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.792042 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rpncv\"" Oct 30 23:57:44.720792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.792179 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lwvlw\"" Oct 30 23:57:44.720792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.792186 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"simulator-w0-d3dcca-kserve-self-signed-certs\"" Oct 30 23:57:44.720792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.792226 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z9rpn\"" Oct 30 23:57:44.720792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.792242 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:57:44.720792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.792287 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.720792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.793665 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7fc6e9-dockercfg-85m2r\"" Oct 30 23:57:44.720792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.793669 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 23:57:44.720792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.793668 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-82d449-openshift-default-dockercfg-mhmr7\"" Oct 30 23:57:44.720792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.793709 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1j1fi1cun0mic\"" Oct 30 23:57:44.720792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.793726 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-59scr\"" Oct 30 23:57:44.720792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.793760 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1bc837-openshift-default-dockercfg-5htq6\"" Oct 30 23:57:44.721235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.793784 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-lc8kv\"" Oct 30 23:57:44.721235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.793797 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-32732d-dockercfg-rcltc\"" Oct 30 23:57:44.721235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.795408 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qmkhv\"" Oct 30 23:57:44.721235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.795432 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.721235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.795492 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:57:44.721235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.795530 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 23:57:44.721235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.795555 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:57:44.721235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.796600 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-e15304\"" Oct 30 23:57:44.721235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.796631 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rb9gl\"" Oct 30 23:57:44.721235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.796808 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 23:57:44.721235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.797115 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-843d8c-dockercfg-ptlb9\"" Oct 30 23:57:44.721235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.798381 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:57:44.721752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.798559 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a63288-serving-cert\"" Oct 30 23:57:44.721752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.798603 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zcrv2\"" Oct 30 23:57:44.721752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.798616 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2mrzn\"" Oct 30 23:57:44.721752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.798629 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 23:57:44.721752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.798645 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 23:57:44.721752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.798670 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lzqxs\"" Oct 30 23:57:44.721752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.798702 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 23:57:44.721752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.798739 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.721752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.799647 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"default-dockercfg-l9wjz\"" Oct 30 23:57:44.721752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.799665 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-anck8g3riltac\"" Oct 30 23:57:44.721752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.800102 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n9xrj\"" Oct 30 23:57:44.721752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.800146 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:57:44.722288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.800167 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"default-dockercfg-75htb\"" Oct 30 23:57:44.722288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.800150 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f026c1-openshift-default-dockercfg-6hgm8\"" Oct 30 23:57:44.722288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.800150 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 23:57:44.722288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.800196 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 23:57:44.722288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.800148 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-a3e876-dockercfg-2tm45\"" Oct 30 23:57:44.722288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.800206 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 23:57:44.722288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.800246 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9tmfn\"" Oct 30 23:57:44.722288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.800932 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 23:57:44.722288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.801698 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"default-dockercfg-jtbmr\"" Oct 30 23:57:44.722288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.801712 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:57:44.722288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.803342 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7fc6e9-serving-cert\"" Oct 30 23:57:44.722288 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.803347 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:57:44.722811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.803381 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"simulator-w2-9ef03a-kserve-self-signed-certs\"" Oct 30 23:57:44.722811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.803440 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-23f75e-openshift-default-dockercfg-ngphj\"" Oct 30 23:57:44.722811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.803444 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7fh9lroec4t9q\"" Oct 30 23:57:44.722811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.804788 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qwtvf\"" Oct 30 23:57:44.722811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.804785 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 23:57:44.722811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.805016 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:57:44.722811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.805037 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 23:57:44.722811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.805024 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-82xnp\"" Oct 30 23:57:44.722811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.806343 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 23:57:44.722811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.806526 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 23:57:44.722811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.806543 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:57:44.722811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.811567 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 23:57:44.723328 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:55:42.823802 7570 logging.go:55] [core] [Channel #2875 SubChannel #2876]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.152:50052", ServerName: "10.134.0.152:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:54862->10.134.0.152:50052: use of closed network connection" Oct 30 23:57:44.723328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:42.946523 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m7.468580878s: [/var/lib/containers/storage/overlay/128bb9236bde63752202e0b28654b6d1251c735561472994283e7a9ccbde243e/diff /var/log/pods/openshift-ingress_e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp_cb95af9b-7595-4dd9-839a-e698e2a22c07/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 23:57:44.723328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.143865 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.723328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.177051 7570 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.723328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.177136 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.723570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.184619 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:57:44.723570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.281551 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.723570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.349586 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.723570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.757013 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 23:57:44.723570 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.757013 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:57:44.723811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.757032 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.723811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.757013 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.723811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.757204 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:57:44.723811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.757207 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:57:44.723811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.757543 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.757555 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.758250 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.758534 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.758727 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.758878 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.759050 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:57:44.724263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.760413 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:57:44.724263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.760581 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.760591 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.760692 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.724513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.760730 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.802593 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.724513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:43.802659 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.155703 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.724513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.155759 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724513 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:55:44.361456 7570 kubelet.go:2545] "Skipping pod synchronization" err="container runtime is down" Oct 30 23:57:44.724766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.760144 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 23:57:44.724766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.760202 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.760376 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.760564 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.760664 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.760340 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.724979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.760835 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 23:57:44.724979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.760932 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:57:44.724979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.761000 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:57:44.724979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.761047 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:57:44.724979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.761106 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.761166 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.761548 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.761673 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.725235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.761706 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.725235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.816909 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.725451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:44.816961 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.725451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.761507 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.761647 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.761809 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.725451 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.761911 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.725686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.761968 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.725686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.762069 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.762119 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.725686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.762174 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.762229 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.762255 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.762284 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.762311 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.762425 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.761777 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:57:44.725903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.762731 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.726169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.764362 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.726169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.783803 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.726169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.783865 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.726169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.803543 7570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-xsv62 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.726169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.803589 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podUID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.726390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:04.904786 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 23:57:44.726390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:05.004988 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.726390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:05.053317 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:57:44.726390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:05.190960 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:57:44.726390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:05.320403 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:57:44.726390 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:56:05.320501 7570 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime is down, PLEG is not healthy: pleg was last seen active 3m14.746726421s ago; threshold is 3m0s]" Oct 30 23:57:44.726690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:05.320619 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": context deadline exceeded" Oct 30 23:57:44.726690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:04.904747 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 23:57:44.726690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:05.442061 7570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-xsv62 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.726690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:05.442127 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podUID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.726690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:05.442256 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:57:44.726690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:05.442552 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": context deadline exceeded" Oct 30 23:57:44.726944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:05.442666 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.726944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:05.701807 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-p5xwl" podUID="ee540c2e-f028-44ee-be53-d178b286fb02" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.24:9403/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.726944 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:56:10.776362 7570 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime is down, PLEG is not healthy: pleg was last seen active 3m20.20259506s ago; threshold is 3m0s]" Oct 30 23:57:44.726944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:22.243065 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Oct 30 23:57:44.726944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:23.685084 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.726944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:25.398309 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.727216 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:25.761659 7570 trace.go:236] Trace[887502929]: "Calculate volume metrics of dshm for pod e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" (30-Oct-2026 23:56:18.154) (total time: 7607ms): Oct 30 23:57:44.727216 ip-10-0-137-81 kubenswrapper[7570]: Trace[887502929]: [7.607062348s] [7.607062348s] END Oct 30 23:57:44.727216 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:25.797708 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 23:57:44.727216 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:27.145653 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:57:44.727216 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:29.002348 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:57:44.727216 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:21.975282 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.727216 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:29.836096 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": context deadline exceeded" Oct 30 23:57:44.727568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:29.989835 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.727568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:30.118195 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.727568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:30.118220 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": context deadline exceeded" Oct 30 23:57:44.727568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:27.299801 7570 patch_prober.go:29] interesting pod/network-check-target-rhm2r container/network-check-target-container namespace/openshift-network-diagnostics: Readiness probe status=failure output="Get \"http://10.134.0.4:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.727568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:22.074281 7570 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-g86k2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9443/health\": context deadline exceeded" start-of-body= Oct 30 23:57:44.727790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.620127 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:57:44.727790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.966756 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.727790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967193 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ww7vc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.727790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967280 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-82d449\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.727790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967338 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.728453 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967381 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.728453 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967419 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k8dwv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.728453 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967456 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.728453 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967516 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.728453 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967556 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.728453 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967732 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z9rpn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.728800 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967801 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.728800 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967847 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.728800 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967901 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7fc6e9-dockercfg-85m2r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.728800 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967943 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"default-dockercfg-75htb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.728800 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967980 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.728800 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968038 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968077 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968114 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968151 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-df3b5f-openshift-default-dockercfg-hmtcd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968187 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-x7g2z\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968225 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968263 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rpncv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729319 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968301 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729319 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968337 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-611ef3-openshift-default-dockercfg-gcsfx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729319 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968391 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729319 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968441 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7fc6e9-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729319 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968493 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2b4009-openshift-default-dockercfg-mnjpt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729319 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968531 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968566 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968603 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hgj94\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968639 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968674 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-anck8g3riltac\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968710 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n9xrj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968746 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968780 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968841 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968876 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7b0299\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968911 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.729854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968946 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"premium-w3-b567bb-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.968981 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969016 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969051 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969085 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qwtvf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969123 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969163 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a63288-dockercfg-5m62n\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969188 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969236 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5f557\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969273 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xj5wh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969308 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969344 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969377 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f026c1-openshift-default-dockercfg-6hgm8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969411 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"default-dockercfg-jtbmr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969444 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969497 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969543 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qbh58\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.969577 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-6c5084-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730608 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.967593 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970153 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"premium-w4-843d8c-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970192 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2mrzn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970238 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970277 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970337 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.730863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970373 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-lc8kv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731118 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970412 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-86d846-dockercfg-dc664\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731118 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970449 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-23f75e-dockercfg-w5fh9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731118 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970505 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731118 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970542 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731118 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970576 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-n7mcg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731118 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970613 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-23f75e\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970647 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970683 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dd3a15-openshift-default-dockercfg-72cjz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970903 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kc52k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970945 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"default-dockercfg-ftrjj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.970990 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-a3e876-dockercfg-2tm45\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731660 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971252 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731660 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971324 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731660 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971363 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-885999-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731660 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971401 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731660 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971436 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731660 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971488 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"simulator-w0-d3dcca-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971526 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1bc837-openshift-default-dockercfg-5htq6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971561 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971596 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971632 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2ccd42-openshift-default-dockercfg-qcc6m\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971668 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.731950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971704 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-df3b5f\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971741 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971776 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971813 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971848 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qmkhv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971884 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732217 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971920 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971955 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jt8kr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.971990 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ttclb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.972024 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.972061 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b567bb-openshift-default-dockercfg-wvjgm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.972095 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-200408\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732850 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.972130 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732850 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.972165 7570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732850 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.972199 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-e15304-dockercfg-8mxkq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732850 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.972798 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a63288-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732850 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.972842 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.732850 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.972878 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"distinct-2-w0-d3dcca-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.972914 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.972949 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.016602 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.016677 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d3dcca-dockercfg-vk8p4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.016721 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.016759 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.016799 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.016841 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"distinct-2-w5-dd3a15-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.016884 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-e15304-openshift-default-dockercfg-qbhbd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.016925 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.016964 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xmrvg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733368 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017002 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017043 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017080 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-843d8c-dockercfg-ptlb9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017118 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017159 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017202 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lzqxs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017240 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-23f75e-openshift-default-dockercfg-ngphj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017282 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017322 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017359 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-611ef3\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017398 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017436 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.733909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017497 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017539 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"default-dockercfg-rzhfh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017577 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-vf4pc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017615 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4bbvk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017655 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fl65j\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017695 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017731 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017768 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017806 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017855 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017891 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gskv9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017929 7570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734417 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.017965 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734697 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.018004 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"distinct-2-w4-843d8c-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734697 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.018041 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"default-dockercfg-l9wjz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734697 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.018078 7570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734697 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.018117 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-885999\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734697 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.018155 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734697 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.018198 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.018234 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.018269 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-v5mxr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.018309 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.018344 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.734959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:28.703588 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.734959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.084694 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.735207 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:28.888630 7570 trace.go:236] Trace[421225445]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" (30-Oct-2026 23:56:24.106) (total time: 4782ms): Oct 30 23:57:44.735207 ip-10-0-137-81 kubenswrapper[7570]: Trace[421225445]: [4.782166716s] [4.782166716s] END Oct 30 23:57:44.735207 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.140823 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.735207 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:29.379886 7570 trace.go:236] Trace[1541388153]: "Calculate volume metrics of iptables-alerter-script for pod openshift-network-operator/iptables-alerter-9nwfw" (30-Oct-2026 23:56:23.616) (total time: 5763ms): Oct 30 23:57:44.735207 ip-10-0-137-81 kubenswrapper[7570]: Trace[1541388153]: [5.76307758s] [5.76307758s] END Oct 30 23:57:44.735207 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:29.867161 7570 trace.go:236] Trace[512339807]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" (30-Oct-2026 23:56:23.376) (total time: 6491ms): Oct 30 23:57:44.735207 ip-10-0-137-81 kubenswrapper[7570]: Trace[512339807]: [6.49111654s] [6.49111654s] END Oct 30 23:57:44.735207 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:29.867179 7570 trace.go:236] Trace[616715948]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" (30-Oct-2026 23:56:23.807) (total time: 6060ms): Oct 30 23:57:44.735207 ip-10-0-137-81 kubenswrapper[7570]: Trace[616715948]: [6.060139148s] [6.060139148s] END Oct 30 23:57:44.735207 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.216981 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.735669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.217040 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7fc6e9\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.735669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.217078 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lwvlw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.735669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.217110 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-82d449-openshift-default-dockercfg-mhmr7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.735669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.217143 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.735669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.217175 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2nz9z\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.735894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.217207 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-200408-dockercfg-zpwrj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.735894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.217250 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-32732d-openshift-default-dockercfg-fvnxc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.735894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.217283 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.735894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.217317 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2b4009\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.735894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.217353 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.735894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.217384 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.736151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.220575 7570 trace.go:236] Trace[610075058]: "Calculate volume metrics of istio-envoy for pod openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" (30-Oct-2026 23:57:00.731) (total time: 1488ms): Oct 30 23:57:44.736151 ip-10-0-137-81 kubenswrapper[7570]: Trace[610075058]: [1.488819266s] [1.488819266s] END Oct 30 23:57:44.736151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.557381 7570 trace.go:236] Trace[1245501656]: "Calculate volume metrics of prometheus-k8s-rulefiles-0 for pod openshift-monitoring/prometheus-k8s-0" (30-Oct-2026 23:56:58.960) (total time: 1760ms): Oct 30 23:57:44.736151 ip-10-0-137-81 kubenswrapper[7570]: Trace[1245501656]: [1.760090773s] [1.760090773s] END Oct 30 23:57:44.736151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.558294 7570 trace.go:236] Trace[923408533]: "Calculate volume metrics of node-exporter-textfile for pod openshift-monitoring/node-exporter-hq9sq" (30-Oct-2026 23:57:00.956) (total time: 1601ms): Oct 30 23:57:44.736151 ip-10-0-137-81 kubenswrapper[7570]: Trace[923408533]: [1.601963733s] [1.601963733s] END Oct 30 23:57:44.736151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.558315 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a63288\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.736151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.558342 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.736151 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.570583 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" podUID="12efb180-e881-40a3-9001-8a39a2ddd352" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.163:50052: i/o timeout" Oct 30 23:57:44.736564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.570829 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 23:57:44.736564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.571278 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.736564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.571357 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" containerName="network-check-target-container" probeResult="failure" output="Get \"http://10.134.0.4:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.736564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.572543 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" podUID="5244915c-9573-489c-b1f8-e22948899df3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.162:50052: i/o timeout" Oct 30 23:57:44.736564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.573298 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.736785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.573331 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-6c5084-openshift-default-dockercfg-bblz7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.736785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.573346 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"premium-w0-d3dcca-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.736785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.392779 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.736785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:01.392903 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" podUID="9c20fb11-2dcf-445e-96cd-4d97045a2ab9" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.15:9443/health\": context deadline exceeded" Oct 30 23:57:44.736785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.575448 7570 trace.go:236] Trace[604143629]: "iptables ChainExists" (30-Oct-2026 23:54:35.929) (total time: 145692ms): Oct 30 23:57:44.736785 ip-10-0-137-81 kubenswrapper[7570]: Trace[604143629]: [2m25.692755727s] [2m25.692755727s] END Oct 30 23:57:44.736785 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.577487 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.794619 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Oct 30 23:57:44.737078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.794674 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" podUID="7f016eec-7091-4a3e-847e-7a382085ffef" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.43:50052: i/o timeout" Oct 30 23:57:44.737078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.794816 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-p5xwl" podUID="ee540c2e-f028-44ee-be53-d178b286fb02" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.24:9403/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.737078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.795006 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:57:44.737078 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:02.795285 7570 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime is down, PLEG is not healthy: pleg was last seen active 3m32.823573392s ago; threshold is 3m0s]" Oct 30 23:57:44.737078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.795455 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded" Oct 30 23:57:44.737327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.802228 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": context deadline exceeded" Oct 30 23:57:44.737327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.802409 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.45:50052: i/o timeout" Oct 30 23:57:44.737327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.802595 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" podUID="a4acfbb3-3ee2-4890-984a-2fc8a796bece" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.158:50052: i/o timeout" Oct 30 23:57:44.737327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:56:21.898147 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.737327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.830911 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-843d8c-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.830972 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1bc837-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.830998 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.831031 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.831058 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-a3e876\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.831091 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.831116 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"distinct-w4-843d8c-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.831140 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.831165 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.831193 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.831232 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-df3b5f-dockercfg-rnjbv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.831271 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d3dcca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.831315 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wczpt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.737854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865023 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.150611931s: [/var/lib/containers/storage/overlay/b11cc6bca26482597c24c4e75e72edf923855dc8db7f34b3289472a8e3ae1c59/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.738123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865021 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.150606182s: [/var/lib/containers/storage/overlay/4b854bec1dba6649d31a3dff052fa6b8475613425b5d06231467ce5435852f05/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.738123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865021 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.150608989s: [/var/lib/containers/storage/overlay/fa20bee63cf629d4324796d5dc18629b47d2d8c07975c10a454de1437a1e7aa8/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.738123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865066 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.150655346s: [/var/lib/containers/storage/overlay/bd64c2e71087c06682d014945c4f9a83c547688a3ba822cda23a6a08d6c484a1/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.738123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865076 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.150662056s: [/var/lib/containers/storage/overlay/24975e45960e735faf152b67e4e3d57c07a6c59fa02b9891c80f4dad14df0448/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.738123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865104 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.150228748s: [/var/lib/containers/storage/overlay/d2d065b7e174e439a27d86e3b269b78b06e80faa352305e8bc9491d4f428d035/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.738123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865100 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.738376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865113 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.150235728s: [/var/lib/containers/storage/overlay/b4092aa5d2eeecf888ef63e1d8f29c78545760cf40bffd8cf15c8bb74a0decd5/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.738376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865121 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-32732d-dockercfg-rcltc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.738376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865143 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.150263643s: [/var/lib/containers/storage/overlay/1ef75a8ed5ab38a35a72a5768007686c92ae81418e40b004b7b856ab5d301165/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.738376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865169 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.150294511s: [/var/lib/containers/storage/overlay/cedc84ddfb5da05d6728336f2b94e22d5afa76627fd24632923cce4f1190193b/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.738376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865257 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.15037783s: [/var/lib/containers/storage/overlay/f236b901b18be822152c6c8bebafe441af5461e6105dd29741cafa752b838e0b/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.738376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865363 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.149502588s: [/var/lib/containers/storage/overlay/378ef04e65b7c86c840fd91c2cf1ea7b4a7c407e830d266802a402fc354353e4/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.738704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.865392 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.149532471s: [/var/lib/containers/storage/overlay/c87476079796d311df4f2d7bb830a092f2d5c1600b87b655e1602a496fc9f961/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.738704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.869748 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.738704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.869808 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"distinct-2-w6-e15304-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.738704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.869854 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.738704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.869890 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-82xnp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.738704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.869926 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.738954 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.869961 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-843d8c\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.738954 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870006 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.738954 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870042 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.738954 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870078 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.738954 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870111 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870143 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-82d449-dockercfg-fwwd9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870166 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.210936448s: [/var/lib/containers/storage/overlay/3d4771aeb60e85fec65b07e561834bcbadbe702e861546262a8b9466d98f9e46/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.739175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870179 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870216 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f0817c\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870250 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870281 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-1j1fi1cun0mic\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870315 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870348 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870390 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"default-dockercfg-4fxkj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870435 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7fh9lroec4t9q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870488 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-843d8c-openshift-default-dockercfg-l4st5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739718 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870520 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rb9gl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739718 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870602 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-a3e876-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739718 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870637 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739718 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870671 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739718 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870705 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739718 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870749 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870781 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.870858 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zcrv2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871214 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9tmfn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871281 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"simulator-w2-9ef03a-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871343 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.739984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871369 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740248 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871394 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740248 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871419 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740248 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871445 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-e15304\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740248 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871490 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-86d846\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740248 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871557 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9ef03a-openshift-default-dockercfg-mf4wj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871622 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871676 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871705 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"default-dockercfg-mdbds\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871747 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871787 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871827 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"simulator-w1-611ef3-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871866 7570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871904 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871948 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.871990 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-885999-dockercfg-tmfpm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872034 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.740811 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872073 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-9ef03a-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872113 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872152 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872189 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-611ef3-dockercfg-m9bvp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872230 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872270 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872306 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872347 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872392 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872442 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872807 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872854 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872887 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872933 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872966 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8gxz9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.872999 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7b0299-openshift-default-dockercfg-cq9kk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873061 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-32732d\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873100 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8bg6s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873137 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d3dcca-openshift-default-dockercfg-mtkk6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873168 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a63288-openshift-default-dockercfg-tdk54\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873223 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-59scr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873256 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873290 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.741831 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873319 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.742085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873350 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.742085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873410 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-2q87t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.742085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873444 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.742085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873544 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hc9wr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.742085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873577 7570 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-djdr2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.742085 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:02.873661 7570 log.go:32] "ListImages with filter from image service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" filter="" Oct 30 23:57:44.742085 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:02.873729 7570 kuberuntime_image.go:108] "Failed to list images" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Oct 30 23:57:44.742383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.873749 7570 image_gc_manager.go:231] "Failed to update image list" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Oct 30 23:57:44.742383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.883671 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.089035443s: [/var/lib/containers/storage/overlay/6eb16d61180a16d56b421e420725672c9540a33d85e4098b7382a9522d59905c/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.742383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.883692 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.117015824s: [/var/lib/containers/storage/overlay/474a9d0694eaae7d10a126a3fdc438562c9207c2be0db9868156f76d5e93a76a/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.742383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.955361 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.240952127s: [/var/lib/containers/storage/overlay/a0083eb3d9103d30009b364e78c040c9b52f64998ca8a845bc87ee92cdc39643/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.742383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.975949 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.073304471s: [/var/lib/containers/storage/overlay/2346d8d60b6d752d20206b66a13de33394b7b349ac3cf5d410a2ef3fc2214ff7/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.742383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.980189 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.082232369s: [/var/lib/containers/storage/overlay/494f889ed424fb93575e03fd6dfb34d2d63211134560d5d81e7863454370da4c/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.742383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.981177 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.742717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.981476 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.742717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.983251 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.742717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.986419 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.272014024s: [/var/lib/containers/storage/overlay/be955eb54abbe2b1ab1729ad2779ccc3080ac5870dec9f0f0d8e5f162b4ab12f/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.742717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.986760 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-89shx" podUID="d3eeb220-e1ba-46e2-94c8-2bb19e02dde9" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.742717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.986974 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": context deadline exceeded" Oct 30 23:57:44.742931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.987020 7570 patch_prober.go:29] interesting pod/dns-default-h4hmx container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.134.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.742931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.987038 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-h4hmx" podUID="cb76bb5e-c0a6-4a38-8732-40193dd39cac" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.742931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.987212 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.742931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.987376 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.742931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.995529 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded" start-of-body= Oct 30 23:57:44.742931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.995563 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded" Oct 30 23:57:44.743180 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.997570 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.743180 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:02.997602 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.743180 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:55:45.762592 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.743180 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.581808 7570 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-g86k2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.743180 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.583360 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 23:57:44.743419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.645274 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.337740315s: [/var/lib/containers/storage/overlay/bba6625d7bd500d93dc1ed92064604daff63f485fb138e04e8c58084b08fb286/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2_131867e6-7957-48d7-96d5-50f9e8ff45b2/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.743419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.645311 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.320347161s: [/var/lib/containers/storage/overlay/792adc4a646bbf2e22f785ca95dead4522d7f98381f9398e19b12c532d7adc97/diff /var/log/pods/openshift-ingress_e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd_a74b8dd0-7067-44fd-94c0-948568ca285c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.743419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.645326 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.037066957s: [/var/lib/containers/storage/overlay/35271418e1c37de4b7888549ea8717976558ef1cb86ef91daf881eb4bb08bde8/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.743419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.645339 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.037078198s: [/var/lib/containers/storage/overlay/0bd751342d0b910bc952cc203f08c3bb411454f4fe1deb6fc808e804fda4ade0/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.743767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.645353 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.037084031s: [/var/lib/containers/storage/overlay/aead9544bd40cc159d1e6a7d220f9f1cabe7f12e5883de5c718c7299e4a254c0/diff /var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/prom-label-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.743767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.645366 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.320183459s: [/var/lib/containers/storage/overlay/d91c0104ace8b482e2dad547d0f33bacdc7842fbd1c6fb1ccd958fb5a4d16f83/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-7rp6c_e7406dbc-b7ad-4b0c-ac37-c85b302bf76e/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.743767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.645378 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.323236468s: [/var/lib/containers/storage/overlay/c9e0b4b340c24fa81a582eac02eb31743c12ef9f25efbd3e606ccfab05aed162/diff /var/log/pods/openshift-ingress_e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24_b94094c6-876a-4082-ad7d-f35bb1909f87/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.743767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.645391 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.319295701s: [/var/lib/containers/storage/overlay/c28f4e8a2c51f59ca420e282df53a9df60396e35d9f7da10932335d0f3b91e34/diff /var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx_dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.743950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.645406 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.036957645s: [/var/lib/containers/storage/overlay/f2c679e7cd5f2bfc6b2642f521367d05b8cdda5b25d215a32900f18b69b6ece9/diff /var/log/pods/openshift-multus_network-metrics-daemon-jgq5s_8cad31bd-cac3-44ca-bb2c-ca302c4e82e4/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.743950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.645429 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.625686011s: [/var/lib/containers/storage/overlay/669c018af63462489022f768fd3845fd1ea75ec4fa3bdc5917e3927d1fed1d30/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.743950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.645451 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.625706592s: [/var/lib/containers/storage/overlay/5e5e4ebc24fc6db62f732ae3ea912af2d07e58df14d44ce8bbb21e2899ed0a4e/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.743950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.645515 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.613943895s: [/var/lib/containers/storage/overlay/e76fa96356ab2d74ab383725a3dc9fb193c9992a8951b183e6a7512a8d67f527/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.743950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646591 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.32475259s: [/var/lib/containers/storage/overlay/d48b62b69845920830050d5e6fd4dfcef959e40ce97b979c3b7bfd3b7240198d/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp_e0f6ac75-7dd2-44be-8b54-92ff018b8a3d/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646614 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038161707s: [/var/lib/containers/storage/overlay/96eefe0d9887665678dd4d2c31afd891226e59681f332d9aefcab1d5c58a06e5/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm_1e8a6ea5-1c91-462f-94fd-7136a48971e9/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646628 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.324646444s: [/var/lib/containers/storage/overlay/5953d0ad13461ca6560c0d4709522e12222222fd97e9131178fdf34eee1fb1f1/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx_6843e367-692b-41f0-a9c6-7704d732f009/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646642 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.320897511s: [/var/lib/containers/storage/overlay/a516443d85784522e54f82a9309c805882c7a29c08ef516c769fec340092039a/diff /var/log/pods/e2e-models-e2e-worker-w2-9ef03a_simulator-w2-9ef03a-kserve-cdb8584b5-76gjs_223c27af-605a-42a8-a29c-ae1fa2b2ace6/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646655 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.323059581s: [/var/lib/containers/storage/overlay/cd3058ba6d2cf52b3aeb9de37169925252ba5677fc23be231ae11bff86ef11a3/diff /var/log/pods/e2e-models-e2e-worker-w4-843d8c_premium-w4-843d8c-kserve-7844d66896-27fxd_4f1bda07-310d-43a8-a926-9004487b0034/main/0.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.744350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646668 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.324989959s: [/var/lib/containers/storage/overlay/44b0704d11c7dc38b323f41e81949db45c4189a4af6346a34d0c7c7250d7f222/diff /var/log/pods/e2e-models-e2e-worker-w3-b567bb_premium-w3-b567bb-kserve-586f4c9b9-rn44f_ed4e6002-31dc-4eb3-9988-d808eee76c7f/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646686 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.321883205s: [/var/lib/containers/storage/overlay/fa06a709a51529569c198f78feb00dfa6f88a021a5da5689e1edbd6007e984a0/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx_cc2a6d9c-ec3e-438a-9d6b-db75fa30a903/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646701 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038226888s: [/var/lib/containers/storage/overlay/1fd92a325a73f706efe6ce154a5764ec96d70f811c06fd3b10df36b85aaaa079/diff /var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal_70da0cbd0028ec04eeaec823ad5f3973/haproxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646724 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038248087s: [/var/lib/containers/storage/overlay/39e54a0677170c09896d6f03393ffd52de086d46bda748378b344b5506f2b596/diff /var/log/pods/openshift-monitoring_telemeter-client-976bbf6cb-lk9st_d0b4de16-5f57-44fe-a273-a77ebd48d050/telemeter-client/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646739 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038261541s: [/var/lib/containers/storage/overlay/e68cb84172be4a39333cd7abcdfcd538da59c3e7519e017364f0f7c93ea51c65/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646754 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038274863s: [/var/lib/containers/storage/overlay/e402921a279de129be7427fee7b99527e8a0e21029c9e1a5f9fbb74cfcd1e91d/diff /var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/thanos-query/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646774 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038294074s: [/var/lib/containers/storage/overlay/9be523497e0fe151cfc66edcd237c2e48a0d02540d5fba80d676ec78482569c5/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/kube-rbac-proxy-metric/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646790 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038308733s: [/var/lib/containers/storage/overlay/75f0108cf238c31c0bcd6dfa87947c79b29edd38a4eeee18c9d56aa443c060c0/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/kube-rbac-proxy-thanos/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646803 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038319876s: [/var/lib/containers/storage/overlay/a644fda2555618d5eec515891b49f1c6f690b61a09095fd4f7c8337e3aff41cd/diff /var/log/pods/kube-system_konnectivity-agent-89shx_d3eeb220-e1ba-46e2-94c8-2bb19e02dde9/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646816 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.32123001s: [/var/lib/containers/storage/overlay/63747dd0209d2941b1b813e2a90b60760a4d2fd53d2e0f2bee93209ea8add28b/diff /var/log/pods/e2e-models-e2e-worker-w0-d3dcca_simulator-w0-d3dcca-kserve-74cd68d695-9s4b4_08cd1e28-c399-4cba-951e-0210a978109f/main/0.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.744790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646828 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.03834197s: [/var/lib/containers/storage/overlay/cf678d272fc7976b06627168ba82c61f095fb38590fbddc535232b9343eb059f/diff /var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646841 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.325498863s: [/var/lib/containers/storage/overlay/e5883ba613520bba70ea60e98be6000acd1a7c947b0f509e4ea4e80af26bcf77/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp_522aa71b-47e4-44b1-825c-bf81305d21b4/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646855 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038364835s: [/var/lib/containers/storage/overlay/d3e41270e106e2e9755a87e8c5fb22722dc048a67ed8221a5648c22d3984b3b1/diff /var/log/pods/e2e-models-e2e-worker-w6-e15304_distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl_ee6148fe-e516-492b-b5a7-755a75e82c58/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646868 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.322479646s: [/var/lib/containers/storage/overlay/2d62442eabbf1ded93ad20b55998de2371ea1396cc7ac9f43a1a9bb40b0f8c27/diff /var/log/pods/openshift-ingress_e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx_f68436c1-d900-4063-b853-cb9be78ceae6/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646887 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038394071s: [/var/lib/containers/storage/overlay/ccf9cc9a8f0788e3f609bf98fb0a6f3bd51c7847ffa988af8d88613a99b8d9ba/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/thanos-sidecar/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646900 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.324408405s: [/var/lib/containers/storage/overlay/ceb920e1e7a3e793f85b86c40fbd5f83c50e42cd965ae018e76950fe864d5206/diff /var/log/pods/openshift-ingress_e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b_cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.744964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646914 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.322271486s: [/var/lib/containers/storage/overlay/9f074815449dfeb2c6adcd6fdde2b1517ee82a0a4d04d335711556b364004ffb/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc_12efb180-e881-40a3-9001-8a39a2ddd352/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.745141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646933 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.03843656s: [/var/lib/containers/storage/overlay/204242ae1cde4dd33bdfafc786b22ed5c635f3c1229b2e586d1533433e1a1de8/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-t49gb_68aefc1f-0a32-4722-a494-054f46f44b5b/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646948 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.03893712s: [/var/lib/containers/storage/overlay/128bb9236bde63752202e0b28654b6d1251c735561472994283e7a9ccbde243e/diff /var/log/pods/openshift-ingress_e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp_cb95af9b-7595-4dd9-839a-e698e2a22c07/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646961 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.325445362s: [/var/lib/containers/storage/overlay/2571647ccfb8952151e56eaff4f7dff6e45bc09d764392a89769ca755476a238/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg_37d7f8cf-f311-4b98-a6b7-25a043ac78ae/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646976 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038532778s: [/var/lib/containers/storage/overlay/3e9c1e5d01d16b92e7db38570c8265c37322278731bdbc94c37ce0c007779620/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g_aa08b804-5246-4fb8-a68c-cf7af9ba4a54/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.745312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.646991 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038489185s: [/var/lib/containers/storage/overlay/f3fdf2a3fa513daa7796a64ce8075659af08ad472684f89a30e46ae75d82d011/diff /var/log/pods/openshift-ingress_e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l_cf77418c-72fd-40ad-b389-df9273f36cfc/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647007 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038566208s: [/var/lib/containers/storage/overlay/5dfb9d812d54804edb00ba844f1a5386ca8ec555bcb79c209fbded98159a4438/diff /var/log/pods/openshift-image-registry_node-ca-vqvzf_f5f12d67-0b46-434f-8c41-a230d421eefd/node-ca/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647020 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038516311s: [/var/lib/containers/storage/overlay/a914f565ef768a0a346be25ac688af1bc707b2b0345a7fc6c3969bd8bb3d2dc2/diff /var/log/pods/openshift-ingress_e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx_570b6b11-715d-4a67-9360-78aa45580d57/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647033 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038582233s: [/var/lib/containers/storage/overlay/16e961f567e71eedb110a42b6f947d298dc907a9c18619a15074bb0879ad3f37/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2_b12dc177-11a1-40b0-9fdf-75993ab176e6/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647046 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.321632571s: [/var/lib/containers/storage/overlay/9149486066c386ec10e44139fc13f1e35814bfa8aaf1c609db95aec5b45cc446/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt_1d0b4116-23a4-4074-8dfb-93b7bbf53681/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647059 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.323981563s: [/var/lib/containers/storage/overlay/9f4d56ce97ccbddd884bdaf82ad15694b5ea0de030e70e61bc178b003829a92f/diff /var/log/pods/openshift-ingress_e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp_61c25f85-fbde-4c00-94c9-502637e57ba5/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647074 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038562602s: [/var/lib/containers/storage/overlay/d56d6140bc7d2e5c56253a9225cca73da7b31f755ae2d850dc9134491ab1586b/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-thls4_3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647095 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038582742s: [/var/lib/containers/storage/overlay/a7c0e7c8693fc8a257065b652fd5066c36bccef876a0e04c0ee9f8a197bc99a5/diff /var/log/pods/openshift-ingress-canary_ingress-canary-9tcb2_284dba08-fe12-4942-a5fd-6c1e2f979a17/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647109 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038596152s: [/var/lib/containers/storage/overlay/a49f35122a945905681209db5ea0ab4a8b935a5cb82ae723cc1ff4dd04653201/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/sbdb/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647122 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038676237s: [/var/lib/containers/storage/overlay/7ea5f4fa2a20fc73e73740f5ddd51ebd7b63eb122f5f18031f79f5619b8a64cb/diff /var/log/pods/openshift-ingress_e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69_84a7842f-6043-4b93-ab32-d25e2ba8c5bb/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647135 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038681563s: [/var/lib/containers/storage/overlay/34a96be9ca0d791025cb1b421a2ac627992ca521fb66ed7f31b0143677ead44b/diff /var/log/pods/openshift-ingress_kube-auth-proxy-dfd47b86c-gjwxh_2b7be075-e8bb-4db1-bfab-782276c29b6e/kube-auth-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647148 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038692395s: [/var/lib/containers/storage/overlay/12fbd2a1baf816c08f27674a73b2c44b1b8d54a47b3b85cab8394c7d94dfa7a7/diff /var/log/pods/e2e-models-e2e-worker-w0-d3dcca_premium-w0-d3dcca-kserve-64955fd666-hx4zl_3e57b4d8-7b73-4490-9138-de9401d6fd65/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647161 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038643926s: [/var/lib/containers/storage/overlay/435030370c99fd90348ec3c51bf775f6d3b9c47c677f9838ab182a8edc20e383/diff /var/log/pods/openshift-ingress_e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk_d8235a15-af1b-4332-ac83-1aef5bfb1286/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647175 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038686829s: [/var/lib/containers/storage/overlay/6b150c5edd7b1ca69f5e1b404aa3bcd052e95d83ed98d6bc55e1f5abd1c1bb35/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z_c15181ee-3e9a-40c9-a029-9ee504a971fa/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647188 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038693128s: [/var/lib/containers/storage/overlay/a7aa06682d5ba76031c6af7007cd8eaac1c0af689bcdd8877a5331a56724a289/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5_c070c544-43c5-4796-b385-6c4094a63624/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.745859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647202 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038748734s: [/var/lib/containers/storage/overlay/1d6721638d637bce25faa19702cfca7d58a751db555957aa13c0046620be55b4/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll_a74a48f3-2b68-424b-9489-33e22dab2f2f/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647215 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038758202s: [/var/lib/containers/storage/overlay/eb6ba043451ed0c3e71aa76c7105b75f5e1db582169d62dc2beefb4fcb9b96e1/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg_7f016eec-7091-4a3e-847e-7a382085ffef/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647228 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.999826964s: [/var/lib/containers/storage/overlay/8fcf25bc01cedf84ba899e5c3eb6261f2177d1fa7eb08356011187d35a0f19a9/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/alertmanager/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647247 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038761041s: [/var/lib/containers/storage/overlay/178e54113f69810cf2114c33a77d5d377253a1a1b94133160924ae9ae8a2f352/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl_a4acfbb3-3ee2-4890-984a-2fc8a796bece/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647275 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038781158s: [/var/lib/containers/storage/overlay/3dcc54b8479d324b621d5d8aef12882854d539507de35502e7c8c3f5c9dfb9e7/diff /var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-vnmqd_baaa353f-43a5-43e5-a73c-92d1840ebdae/cert-manager-webhook/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647290 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038835169s: [/var/lib/containers/storage/overlay/a4d268a8a8717d1a62b159ce196714489eee4b826ca22119f35fb77eda7664ca/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb_d387eda1-c364-4443-ba0d-619368f57f28/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647322 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.326193407s: [/var/lib/containers/storage/overlay/daa585bcebbabe4fec4b3941b1eaed6ce79b204a87bc255f0d0c36f7c6397f9b/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-p5kqz_9a41f90f-265b-4a47-a2bc-9239bbd1822b/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647335 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038817124s: [/var/lib/containers/storage/overlay/9640cee11f897f54ce412f9f51a65f38b489af493e772d7cc178bb778d3dc805/diff /var/log/pods/kube-system_global-pull-secret-syncer-6sfch_6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647348 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038849743s: [/var/lib/containers/storage/overlay/8f54da0a78596d861902bd46c804f7cad6acf955f67af06c73b41663d2c38df0/diff /var/log/pods/e2e-models-e2e-worker-w4-843d8c_distinct-w4-843d8c-kserve-df87d4cff-tz6fp_80dc7b17-04ea-4d03-a843-24891d3cda07/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746362 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647362 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038856181s: [/var/lib/containers/storage/overlay/01bc050ff3ede245fc5b6578e08988263fdbbae208afaf2849addc5a2ddefe4b/diff /var/log/pods/openshift-ingress_e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj_4c7f3b9a-978c-49f2-a52a-5293dfb1924f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.746362 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647376 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038876639s: [/var/lib/containers/storage/overlay/e09987724efa3cf21524ab84afb430b55533805bb956a307ab47256cb255761c/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mxggs_363febe9-9102-48c9-8faa-09bf49151bf0/discovery/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746362 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647389 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.931532447s: [/var/lib/containers/storage/overlay/b84a05b52289da70061eec39f056a06299c2328aff3d97d572eba8b93ab97836/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746362 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647406 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038891142s: [/var/lib/containers/storage/overlay/e6f8c59141778017279d386690b2eac72a1855b721e05047a44da311020b5367/diff /var/log/pods/openshift-ingress_e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6_e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647424 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.000022116s: [/var/lib/containers/storage/overlay/1f44acf446b79c4eeef279d5c019e358a07c00b124e437316480cb88324a473c/diff /var/log/pods/openshift-ingress_e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724_d2658215-b85d-44d6-8d22-9803dd36e9f4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647438 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038931092s: [/var/lib/containers/storage/overlay/f0723e62218d24ac3caf102b2bef1d64af6ea4758cdc5b0f62128eef666a2440/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq_201fbcb8-8fed-44e3-b60d-4b89d8c114ca/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647452 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.038433947s: [/var/lib/containers/storage/overlay/e3665a62bfc0dfd6fa6ce6260d72c4e844ec3f4b9ca453eee6cd2c040b5d6087/diff /var/log/pods/openshift-ingress_e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28_ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746605 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647485 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.034572036s: [/var/lib/containers/storage/overlay/1d382d86ece2071185bb89cc0a01ab5774ba0076942f4a8c4e654c71c3ce26c3/diff /var/log/pods/openshift-ingress_e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv_4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647499 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.000091857s: [/var/lib/containers/storage/overlay/fe49837001233bfe83175e53f6914495bbf08c293f12e080bada70b26eadec99/diff /var/log/pods/e2e-models-e2e-worker-w0-d3dcca_distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb_42fdc936-a982-454f-a538-0a6494d1cfa2/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647513 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.460900455s: [/var/lib/containers/storage/overlay/9c08cbdbd940b063d50924e2f8b7c36951b33066cffbbb96219f325d0aed3c22/diff /var/log/pods/openshift-ingress_e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr_f8b86c0c-3f2f-4f9d-be26-948cb9084373/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647527 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.039010724s: [/var/lib/containers/storage/overlay/3aa1b53568922c8825d9dde751c035f12b5eb8ac8bf6695fac5a7451543db9ec/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c_e9c9e5ed-908c-463d-8f4f-395a80ababdd/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647542 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.0390496s: [/var/lib/containers/storage/overlay/c98a22f1fd04912f9efa076a547dd3bd39426a4efe5407ae848bcad8b8f23e08/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght_b4e6db40-bf7c-4440-bed0-8ec6e70cab0f/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.746918 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647556 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.00015193s: [/var/lib/containers/storage/overlay/8549b996ed862bedf8bbe0d0e6d867083d48dff353f38ee03b336ac9bf5c52b6/diff /var/log/pods/openshift-ingress_e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr_1017ca7a-a16e-44bd-af55-d8babaa94e01/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746918 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647575 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693170481s: [/var/lib/containers/storage/overlay/eb10cfe189a72077416d9595382c191d23f6a1ec9520e0c5ebf60082667efb79/diff /var/log/pods/openshift-dns_dns-default-h4hmx_cb76bb5e-c0a6-4a38-8732-40193dd39cac/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746918 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647589 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693184465s: [/var/lib/containers/storage/overlay/32f4ef24912ac2375218657860eb7c8f91cc7af57d07641a0404d19548193274/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/prom-label-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.746918 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647604 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693196302s: [/var/lib/containers/storage/overlay/60fd7647cad01e28ecb0b6fa0b7fb4214c64cceae69cc10da083d8561c386c38/diff /var/log/pods/openshift-monitoring_telemeter-client-976bbf6cb-lk9st_d0b4de16-5f57-44fe-a273-a77ebd48d050/reload/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647618 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693210051s: [/var/lib/containers/storage/overlay/7dae0ecfe0d8187b7dbfbb65052d8195244515a6a820bd37827468962332fc52/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647632 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693222976s: [/var/lib/containers/storage/overlay/47793c876c4c5d01e87c8506c573d65ce5cc0467d2ea9e7fd3d9e89277e28eb8/diff /var/log/pods/openshift-network-diagnostics_network-check-target-rhm2r_a10c4ed2-0361-47e1-8810-8a710b2be42d/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647645 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.039135685s: [/var/lib/containers/storage/overlay/5b90dd098e4d643f1d5a31069cb1bf326cf53938c3f35db3115ff7673a58c617/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8_fb14f0ca-115b-4fe6-afcd-6338df872573/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647658 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693242546s: [/var/lib/containers/storage/overlay/53a2f0ab3acd08f98c4309cd58184f7de269803337be73f113d8fdd5ad4472f1/diff /var/log/pods/openshift-monitoring_metrics-server-5946dc94c7-djbgv_ed360b42-1a02-4474-9bf1-2117d1b438f1/metrics-server/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647671 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693254542s: [/var/lib/containers/storage/overlay/9a9f4be3a49018715bcb4d1c4d5cb65b74468a2e4e310a505a73f5b8e1792030/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647685 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693266496s: [/var/lib/containers/storage/overlay/90efbca770046be08c5d002f48b022752d474c92671e020681e23e6db5e7cd98/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/config-reloader/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647699 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693278705s: [/var/lib/containers/storage/overlay/4d86043d70c001d12673c73c993554093cf74312485ab5672b56588b1fc45be0/diff /var/log/pods/openshift-multus_multus-nkhth_89205e33-cb00-47fe-b78d-cc1a05d1f123/kube-multus/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647719 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.69329377s: [/var/lib/containers/storage/overlay/815810e51a6e87afbade1c836ec9b1910d94fe93ec5545786f97c29233ab3971/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-562ql_19274393-d63a-4889-8c71-0648e21959a5/nbdb/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647734 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693308082s: [/var/lib/containers/storage/overlay/beb2779a43a009c6356e35f50542f5321cac999f5dae351ed0adea127279521f/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/prometheus/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647747 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693319748s: [/var/lib/containers/storage/overlay/620ee42d53049268fd47d0f1a39664ca22d1fdff365003b2296756adf18bac33/diff /var/log/pods/openshift-insights_insights-runtime-extractor-7vq6t_d54e0641-5f0d-41bd-8b1b-900463a24cfa/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647763 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693334606s: [/var/lib/containers/storage/overlay/a404bf6c87b5c53e3878ddb51210498f4557b3e883c2ccd89e7b100ce5a1b5b9/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-thls4_3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/kube-rbac-proxy-self/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647777 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693346622s: [/var/lib/containers/storage/overlay/56baec63f55948cea80d82961f0c5560a1f7f4c57db35b0f4a7c40841e4af8b8/diff /var/log/pods/openshift-monitoring_node-exporter-hq9sq_a0371e4c-ff4b-4867-98a1-bb598ab03845/node-exporter/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647789 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.69335774s: [/var/lib/containers/storage/overlay/a255baffe46fcba829c6110007ac4ed6c37edae172e754e6991d3d93804591bb/diff /var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/kube-rbac-proxy-metrics/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747682 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647802 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.039290969s: [/var/lib/containers/storage/overlay/a28a107902b3427100e50707b355494f8e1097b6f186562f4cb05c85c51bb041/diff /var/log/pods/openshift-ingress_e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk_af49f007-5f21-43cc-b4f3-d78e31b45100/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747682 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647816 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.039300386s: [/var/lib/containers/storage/overlay/3850d1bc8fbc7eb22abcc4a003e89aeb5ab4318b239a2b288cd6c99729884848/diff /var/log/pods/openshift-ingress_e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd_cbf9885e-6fe7-4c33-b91a-60b11bf4b730/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.747682 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647835 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693398841s: [/var/lib/containers/storage/overlay/f973cad01573d376a84630445560caa82c3138f0a6ba74fb9d19bd0b3f3d1fdc/diff /var/log/pods/openshift-ingress_e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798_c9e446a0-c6da-40ab-aa1f-f3756e39619a/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747682 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647863 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693421826s: [/var/lib/containers/storage/overlay/0c09db27888916a85d91c941bc52228061c76fa94b736a8a3dc7e8996ff1a7af/diff /var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647878 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.69347538s: [/var/lib/containers/storage/overlay/fd98498c4d8b710748c4ae2ca27074d51aa81467be4cb8008b2567f3234a8e94/diff /var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5n89j_20fc3e5f-362e-41f7-bb52-2b171d7af818/manager/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647892 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693445607s: [/var/lib/containers/storage/overlay/c4dcae9a167c26250c827408ebc57ee2fa7496cbfcb1dcf9e64c5f3afad71885/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647910 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693462811s: [/var/lib/containers/storage/overlay/48554265d83ff926268710f46b984028e52680fd6021993e647cc43f95b565f4/diff /var/log/pods/openshift-dns_node-resolver-bwf5h_1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.747863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647925 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.761196389s: [/var/lib/containers/storage/overlay/ce21d6da650dbbfd9753673e28424d581b97e8ff8622c073b0da6189d3609be4/diff /var/log/pods/openshift-ingress_e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7_31585445-5877-4947-8df5-f30317948127/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647962 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.932104051s: [/var/lib/containers/storage/overlay/7e97f3d8d436ca6beb296709e5aca670ebcfb0dd4aabd2b68f7b863b769db52b/diff /var/log/pods/openshift-ingress_e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk_51164b20-de0a-4b55-bdb7-7c25c2d5d922/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.647977 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693565473s: [/var/lib/containers/storage/overlay/2b215fe76accc9bae0d0931ffd6d952c738e9fb935e12cf2401502156062262d/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb_bfa36aaf-a967-456c-a544-9a2bcc4ac7d3/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648007 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.039503921s: [/var/lib/containers/storage/overlay/f0631351a60de912d395b3e14337dbdacc4b4f496b2f42a3a01d46349b9e8909/diff /var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2_5275c1e3-85ea-449f-b526-7c3cbb14ed4c/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648025 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693548505s: [/var/lib/containers/storage/overlay/05fd72b3c91f09e3e27d0f3dcbc88b11565ca12103c50c8ba3883064f99b1120/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-t49gb_68aefc1f-0a32-4722-a494-054f46f44b5b/openshift-state-metrics/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748214 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648047 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693568561s: [/var/lib/containers/storage/overlay/f8e3539a2bfe6da4be25bcb30af5428ab1c1d3d6fab81f0b22035429b5c673ee/diff /var/log/pods/openshift-dns_dns-default-h4hmx_cb76bb5e-c0a6-4a38-8732-40193dd39cac/dns/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748214 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648062 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693581534s: [/var/lib/containers/storage/overlay/26ac99fb02a37e4926c08c8653a404ff86c95c4d78e127e481ea02248e842af9/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-qmn89_88de885b-31c5-4ce7-a66f-0a212087affd/tuned/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748214 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648076 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.93222062s: [/var/lib/containers/storage/overlay/30b4a6775958a4ef1a2b395cf67610d8b6c311908f801be25df2cebe10e12d71/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5_e8c5ef3a-fb2d-4f45-9d16-470f553b62c5/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748214 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648089 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693640834s: [/var/lib/containers/storage/overlay/4bcfc2e73e803edc6350b7e0219f16061b0b0154c38ef68fc9b30848eef9d488/diff /var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-68r7m_347b9a5b-729f-419a-bce5-749ac4aca5b5/cert-manager-cainjector/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648102 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693618278s: [/var/lib/containers/storage/overlay/e593b8801075c6881923384cd19c2b28622b2fac411e5f27d5644bb6a7ac66d7/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx_08022df4-0b83-43af-9b14-b85ab9f83be3/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648116 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693678205s: [/var/lib/containers/storage/overlay/6be4816b08509c82430a0d49d48da911e6724b5bacbaa526b682fea5602b8f39/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4_ca8613ba-3272-43d3-9661-33618f859cff/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648129 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.761399216s: [/var/lib/containers/storage/overlay/5208d4cca466afc2340fecbbdbdaf54e0318e89daf9479af1370e69dacb7f283/diff /var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-c6888_eeffad66-ea6b-486f-abd8-63e68120af4c/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648147 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.000739455s: [/var/lib/containers/storage/overlay/ca3f279b0d073526d1fe7e0960f66b4e79def3c3cdd34e0da948566960bdf993/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh_eedd4bdb-1495-47e6-884d-5ce26a05e72b/maas-api/0.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.748601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648163 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.693762217s: [/var/lib/containers/storage/overlay/88cb8fa866137edd068803e272efab84dc735ed9ba27c9971d12ae783f3c7189/diff /var/log/pods/openshift-ingress_e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz_48390cfe-87a3-4b4e-9986-8adec60f6349/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648178 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.69374291s: [/var/lib/containers/storage/overlay/dc96c17fd475929835eebbdb4acc0176ce21564a081f59b7dfbe09fc27de2e46/diff /var/log/pods/e2e-models-e2e-worker-w4-843d8c_distinct-2-w4-843d8c-kserve-8644666f64-vhwnn_d073cc7c-6856-4131-8bf7-250400f05d7f/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648388 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.932526926s: [/var/lib/containers/storage/overlay/bcba7dcacbd58646714997736c6e7a16279f4eee41283f3e635085ad29b7bfff/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5_06e6c870-08ba-4b3c-a26d-45397d5ca730/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648757 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.69433752s: [/var/lib/containers/storage/overlay/52376acc02e0f243a72a19182304da6dbecad3d68ff1fade52ae6da04a76ffc0/diff /var/log/pods/openshift-ingress_e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4_e8fab750-eae1-4f10-b8f3-3db73b00a1f9/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.648880 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.694436774s: [/var/lib/containers/storage/overlay/ec332e0f46b7ba7d9795eb4bdc5cc199e37edddb3407efb9995e04836440258a/diff /var/log/pods/opendatahub_odh-model-controller-769875db56-4blbf_1793bf9b-92db-4dae-bfd6-f86e5b076739/manager/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.951149 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Oct 30 23:57:44.748773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.951586 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 23:57:44.748773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.951862 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.748773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.965118 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" podUID="9c20fb11-2dcf-445e-96cd-4d97045a2ab9" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.15:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.748994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.965931 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.250063354s: [/var/lib/containers/storage/overlay/f39dcaf820ff00166e8c80e107d5fdb36b312bc3fe68e119f0a22871a807ba49/diff /var/log/pods/kuadrant-system_authorino-74dc7574c6-6rws8_3dc85af3-2ef5-4005-a8c5-d515a92d2eee/authorino/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.966100 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.0116538s: [/var/lib/containers/storage/overlay/4b3faf0029ea6a1ccb158e79150f8ca77b731ff0151cd603c0ad850fe13179f9/diff /var/log/pods/openshift-ingress_e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf_f179a0a0-c464-4a8b-b6cc-04a21610a832/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.748994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.966442 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.012006405s: [/var/lib/containers/storage/overlay/c85c869fda9cc50e4cdcac383e3458aacd25870ed3ae9ea9ada931827e9e3928/diff /var/log/pods/openshift-ingress_e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9_29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.748994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.969185 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.014730995s: [/var/lib/containers/storage/overlay/0a161f603ddfa64eb35c23099fb0a2b72d085fea8d70ad02d1375d6d034bb28c/diff /var/log/pods/e2e-models-e2e-worker-w1-611ef3_simulator-w1-611ef3-kserve-8455bfc968-b9llt_00b04bdb-908f-4ced-b3d2-1627d96513b6/main/0.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.749168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.969402 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.082662675s: [/var/lib/containers/storage/overlay/c4f93fe15e5a9ad51f8b72d54bff1b29bbbb4a273ba52fb02b2d86cfb90e177a/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt_7371f1ff-163e-49b2-a4ea-5a9bf3a18215/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.749168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.970430 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.01597261s: [/var/lib/containers/storage/overlay/53894e6b146e5677d226eef0d3d36ba72db36ebdfa61a0477624ed839c3c58dd/diff /var/log/pods/openshift-ingress_e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl_a6f473a6-9517-4367-b935-294dbd28ceec/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.749168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.970754 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.016266488s: [/var/lib/containers/storage/overlay/ac5eaa0944ce2c0dad54c06c9c14404c9a7d378cb21855d1933155f8353a8355/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6986c789f7-5bhpc_d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e/manager/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.749168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.970860 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.016377501s: [/var/lib/containers/storage/overlay/b3a6db58fb09b4d0c2798278ae2643b265542d44375feee3ad946e1ec68a7f42/diff /var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-6pxqs_f87cf2ab-c80c-4680-9c10-497e2a11235b/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.749168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.970912 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:57:44.749383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.970923 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.016522936s: [/var/lib/containers/storage/overlay/1b779171f801079be98b1c067503ccbdf021c24d100122b866226e7349db6559/diff /var/log/pods/cert-manager_cert-manager-597b95cb84-p5xwl_ee540c2e-f028-44ee-be53-d178b286fb02/cert-manager-controller/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.749383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.971145 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.016650388s: [/var/lib/containers/storage/overlay/cc8a1b4516cc9e7213e654a25f68a4ce41b17570a17435d3cbfdaf9510aa01f0/diff /var/log/pods/e2e-models-e2e-worker-w5-dd3a15_distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r_809bef14-259b-4a21-b170-a6a701e00e6c/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.749383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.971141 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 23:57:44.749383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.971181 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:57:44.749383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.971589 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.017095005s: [/var/lib/containers/storage/overlay/70fdb3a403ae61abbfa9d076234bdabd603c276cad5bf4aef6ae75ef402d0cfe/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb_4af4419c-fffd-463e-9577-94bad299abc6/maas-api/0.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.749779 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.971720 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.017230784s: [/var/lib/containers/storage/overlay/69a1b67a43a502a6a2d76337b00d6bf126a091f49f8f5595e1cb19843da575ce/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-t7b8g_02d621d9-9a83-45c7-bef8-9255f543c6f0/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.749779 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.972958 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:57:44.749779 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:03.970850 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded" Oct 30 23:57:44.749779 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.121562 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.167100379s: [/var/lib/containers/storage/overlay/6a8fc60645b598451596d1e8407ac9bf942b9229630a08205b8bbe6683d5e429/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc_6b3d259c-4f6c-4033-b6dd-1059616fe3b4/maas-api/0.log]; will not log again for this container unless duration exceeds 4s Oct 30 23:57:44.749779 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.133599 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.179107122s: [/var/lib/containers/storage/overlay/c08014ec14fb4421165ffa63f45e8f2812548ee47468644b0418f79bc2b0f6ad/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn_2533ebe5-857f-4255-8c85-5a526fe1062b/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.750006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.135418 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.180972673s: [/var/lib/containers/storage/overlay/1108fec57902ce5126303aeb6c6007e464c3a285cfeb7121f254f723b05f1df8/diff /var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-x6pgq_2b47b3cc-9735-45f1-8d87-7edd6ef95dd8/jobset-operator/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.750006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.135435 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.063122243s: [/var/lib/containers/storage/overlay/d42f532082121d22d2dae8c644e5656abc7eae5ca7871bd80e4a5fb632ca6649/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.750006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.245447 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.750006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.260229 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded" start-of-body= Oct 30 23:57:44.750006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.260279 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded" Oct 30 23:57:44.750006 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.260812 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 30 23:57:44.750263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.261338 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:57:44.750263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.261584 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:57:44.750263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.261850 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:57:44.750263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.262063 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 23:57:44.750263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.262402 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 30 23:57:44.750263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.262739 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded" Oct 30 23:57:44.750527 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.262964 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 30 23:57:44.750527 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.263350 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Oct 30 23:57:44.750527 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.263677 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:9091/-/healthy\": context deadline exceeded" start-of-body= Oct 30 23:57:44.750527 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.263715 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/healthy\": context deadline exceeded" Oct 30 23:57:44.750527 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.263753 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": context deadline exceeded" Oct 30 23:57:44.750527 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.263972 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 30 23:57:44.750789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.264426 7570 patch_prober.go:29] interesting pod/metrics-server-5946dc94c7-djbgv container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:10250/livez\": context deadline exceeded" start-of-body= Oct 30 23:57:44.750789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:04.264488 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" podUID="ed360b42-1a02-4474-9bf1-2117d1b438f1" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.14:10250/livez\": context deadline exceeded" Oct 30 23:57:44.750789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.765192 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.750789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.765557 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal" podUID="70da0cbd0028ec04eeaec823ad5f3973" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 23:57:44.750789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.765632 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 23:57:44.750789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.765864 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:57:44.751056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.767239 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.751056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.767911 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:57:44.751056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.768128 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.751056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.769179 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:57:44.751056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.769815 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 23:57:44.751287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.770246 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.751287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.770491 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 30 23:57:44.751287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.770702 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded" Oct 30 23:57:44.751287 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:37.781132 7570 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime is down, PLEG is not healthy: pleg was last seen active 4m47.207371335s ago; threshold is 3m0s]" Oct 30 23:57:44.751287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.793899 7570 ???:1] "http: TLS handshake error from 10.134.0.14:37960: EOF" Oct 30 23:57:44.751287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.810659 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" podUID="6843e367-692b-41f0-a9c6-7704d732f009" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.149:50052: i/o timeout" Oct 30 23:57:44.751287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.815423 7570 ???:1] "http: TLS handshake error from 10.134.0.17:48016: read tcp 10.0.137.81:10250->10.134.0.17:48016: read: connection reset by peer" Oct 30 23:57:44.751287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.815492 7570 ???:1] "http: TLS handshake error from 10.134.0.17:48018: EOF" Oct 30 23:57:44.751739 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:37.819841 7570 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a89e5e1c7169248cead0c4e7f1cca8d1-9b02837b85ac7765.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/e2e-models-e2e-worker-w3-b567bb/events/premium-w3-b567bb-kserve-586f4c9b9-rn44f.18e3732e923c021f\": dial tcp: lookup a89e5e1c7169248cead0c4e7f1cca8d1-9b02837b85ac7765.elb.us-east-2.amazonaws.com: i/o timeout" event="&Event{ObjectMeta:{premium-w3-b567bb-kserve-586f4c9b9-rn44f.18e3732e923c021f e2e-models-e2e-worker-w3-b567bb 45510 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:e2e-models-e2e-worker-w3-b567bb,Name:premium-w3-b567bb-kserve-586f4c9b9-rn44f,UID:ed4e6002-31dc-4eb3-9988-d808eee76c7f,APIVersion:v1,ResourceVersion:31935,FieldPath:spec.containers{main},},Reason:Unhealthy,Message:Liveness probe failed: Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers),Source:EventSource{Component:kubelet,Host:ip-10-0-137-81.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:42:00 +0000 UTC,LastTimestamp:2026-10-30 23:53:24.347408416 +0000 UTC m=+2015.866409498,Count:5,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-81.us-east-2.compute.internal,}" Oct 30 23:57:44.751739 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.820972 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.751859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.822743 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.751859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.823041 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.751859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.824099 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.751859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.830640 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.751859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.831139 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.752077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.831769 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.752077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.831990 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.752077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.832274 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.752077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.833344 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.752077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.833393 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.752348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.833475 7570 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.752348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.833508 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.752348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.833557 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.752348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.833590 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.752348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.833647 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.752612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.833669 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.752612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.835031 7570 ???:1] "http: TLS handshake error from 10.134.0.14:53590: EOF" Oct 30 23:57:44.752612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.835033 7570 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.752612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.835082 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.752612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.836571 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.752612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.836616 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.752876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.836673 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.752876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.836696 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.752876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.836949 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.752876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.837053 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 30 23:57:44.752876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.837440 7570 patch_prober.go:29] interesting pod/metrics-server-5946dc94c7-djbgv container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.753100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.837519 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" podUID="ed360b42-1a02-4474-9bf1-2117d1b438f1" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.753100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.838205 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.753100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.843032 7570 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.888528415s: [/var/lib/containers/storage/overlay/ebf739cfecd58d11e198de589f2102d9749ef735c638b4e295e1e748954c54c9/diff /var/log/pods/openshift-ingress_e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq_67262faa-4e17-4e80-9d44-917ef37e70db/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 23:57:44.753100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.845063 7570 patch_prober.go:29] interesting pod/network-check-target-rhm2r container/network-check-target-container namespace/openshift-network-diagnostics: Readiness probe status=failure output="Get \"http://10.134.0.4:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.753100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.845089 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-network-diagnostics/network-check-target-rhm2r" podUID="a10c4ed2-0361-47e1-8810-8a710b2be42d" containerName="network-check-target-container" probeResult="failure" output="Get \"http://10.134.0.4:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.753316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.845439 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-p5xwl" podUID="ee540c2e-f028-44ee-be53-d178b286fb02" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.24:9403/livez\": dial tcp 10.134.0.24:9403: connect: connection refused" Oct 30 23:57:44.753316 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:57:37.845744 7570 logging.go:55] [core] [Channel #2884 SubChannel #2885]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.171:50052: operation was canceled" Oct 30 23:57:44.753316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.846776 7570 patch_prober.go:29] interesting pod/dns-default-h4hmx container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.134.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.753316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.846800 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-h4hmx" podUID="cb76bb5e-c0a6-4a38-8732-40193dd39cac" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.753316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.862175 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-89shx" podUID="d3eeb220-e1ba-46e2-94c8-2bb19e02dde9" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.753316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.903247 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="da3a65c4-b342-42d7-aad0-29e63b90125a" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 23:57:44.753642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.903626 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" podUID="19274393-d63a-4889-8c71-0648e21959a5" containerName="ovnkube-controller" probeResult="failure" output="command timed out" Oct 30 23:57:44.753642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.903766 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" podUID="19274393-d63a-4889-8c71-0648e21959a5" containerName="sbdb" probeResult="failure" output="command timed out" Oct 30 23:57:44.753642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.903937 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="da3a65c4-b342-42d7-aad0-29e63b90125a" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 23:57:44.753642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.904042 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-562ql" podUID="19274393-d63a-4889-8c71-0648e21959a5" containerName="nbdb" probeResult="failure" output="command timed out" Oct 30 23:57:44.753642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.904156 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-7vq6t" podUID="d54e0641-5f0d-41bd-8b1b-900463a24cfa" containerName="extractor" probeResult="failure" output="command timed out" Oct 30 23:57:44.753642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.904572 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: TLS handshake timeout" Oct 30 23:57:44.753642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.906944 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.753948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.907594 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.753948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.909341 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/healthz\": dial tcp 10.134.0.23:8081: connect: connection refused" Oct 30 23:57:44.753948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.909414 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/readyz\": dial tcp 10.134.0.23:8081: connect: connection refused" Oct 30 23:57:44.753948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.927217 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:57:44.753948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.927319 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.753948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.929289 7570 trace.go:236] Trace[1789831648]: "Reflector WatchList" name:object-"opendatahub-operator-system"/"openshift-service-ca.crt" (30-Oct-2026 23:57:02.885) (total time: 35044ms): Oct 30 23:57:44.753948 ip-10-0-137-81 kubenswrapper[7570]: Trace[1789831648]: [35.044027826s] [35.044027826s] END Oct 30 23:57:44.753948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.929372 7570 trace.go:236] Trace[1822846605]: "Reflector WatchList" name:object-"openshift-insights"/"insights-runtime-extractor-sa-dockercfg-z9rpn" (30-Oct-2026 23:57:04.060) (total time: 33868ms): Oct 30 23:57:44.753948 ip-10-0-137-81 kubenswrapper[7570]: Trace[1822846605]: [33.868753297s] [33.868753297s] END Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.929429 7570 trace.go:236] Trace[915844626]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-inf-a-2b4009-openshift-default-dockercfg-mnjpt" (30-Oct-2026 23:57:04.148) (total time: 33780ms): Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: Trace[915844626]: [33.780830149s] [33.780830149s] END Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.932107 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.932970 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.933186 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"default-dockercfg-ftrjj\"" Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.933395 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.933428 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.935777 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.935853 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.935912 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.935936 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.935948 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.754305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.935987 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.754861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936034 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.754861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936088 7570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:57:44.754861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936120 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.754861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936157 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.754861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936181 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d3dcca\"" Oct 30 23:57:44.754861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936225 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:57:44.754861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936218 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 23:57:44.754861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936279 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.754861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936283 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.754861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936035 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f0817c\"" Oct 30 23:57:44.754861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936352 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.754861 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936374 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.755542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936420 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 23:57:44.755542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936517 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.755542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936592 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.755542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.936598 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.755542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.937073 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"premium-w3-b567bb-kserve-self-signed-certs\"" Oct 30 23:57:44.755542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.937199 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.755542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.937619 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:57:44.755542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.938147 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.755542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.938273 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"premium-w0-d3dcca-kserve-self-signed-certs\"" Oct 30 23:57:44.755542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.939311 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.755542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.939383 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-843d8c-openshift-default-dockercfg-l4st5\"" Oct 30 23:57:44.755542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.939452 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 23:57:44.756050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.941336 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.756050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.941423 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-59scr\"" Oct 30 23:57:44.756050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.942397 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:57:44.756050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.942726 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jt8kr\"" Oct 30 23:57:44.756050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.942766 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7b0299-openshift-default-dockercfg-cq9kk\"" Oct 30 23:57:44.756050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.942773 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8bg6s\"" Oct 30 23:57:44.756050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.942813 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:57:44.756050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.942822 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"distinct-2-w5-dd3a15-kserve-self-signed-certs\"" Oct 30 23:57:44.756050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.942726 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"default-dockercfg-jtbmr\"" Oct 30 23:57:44.756050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.942887 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"default-dockercfg-mdbds\"" Oct 30 23:57:44.756050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.942905 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"default-dockercfg-rzhfh\"" Oct 30 23:57:44.756050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.942933 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kc52k\"" Oct 30 23:57:44.756574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.942974 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a63288-serving-cert\"" Oct 30 23:57:44.756574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.943058 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qmkhv\"" Oct 30 23:57:44.756574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.943059 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 23:57:44.756574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.943111 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-611ef3-openshift-default-dockercfg-gcsfx\"" Oct 30 23:57:44.756574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.943178 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:57:44.756574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.943315 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"simulator-w1-611ef3-kserve-self-signed-certs\"" Oct 30 23:57:44.756574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.943816 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 23:57:44.756574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.944050 7570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:57:44.756574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.944160 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-lc8kv\"" Oct 30 23:57:44.756574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.944256 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d3dcca-dockercfg-vk8p4\"" Oct 30 23:57:44.756574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.944353 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k8dwv\"" Oct 30 23:57:44.758803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.944394 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded" start-of-body= Oct 30 23:57:44.758803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.944439 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded" Oct 30 23:57:44.758803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.944655 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 30 23:57:44.758803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.944949 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:57:44.758803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.958942 7570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:57:44.758803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:37.974433 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:57:44.758803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:38.633297 7570 container_manager_linux.go:682] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 35471 into the system container \"/system.slice\": " Oct 30 23:57:44.758803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:38.690577 7570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:57:44.759028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:38.777341 7570 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-g86k2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.759028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:38.777408 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" podUID="9c20fb11-2dcf-445e-96cd-4d97045a2ab9" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.15:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.759028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:38.831971 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.759028 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:38.836744 7570 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee540c2e_f028_44ee_be53_d178b286fb02.slice/crio-conmon-318dfc9987ebf2f548bc401821e12fd61ce5837c82803f1b0ead37e89b9a688b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 23:57:44.759028 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:38.837172 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded" Oct 30 23:57:44.759158 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:57:38.837353 7570 logging.go:55] [core] [Channel #2886 SubChannel #2887]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:52346->10.134.0.171:50052: use of closed network connection" Oct 30 23:57:44.759158 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:38.837960 7570 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee540c2e_f028_44ee_be53_d178b286fb02.slice/crio-conmon-318dfc9987ebf2f548bc401821e12fd61ce5837c82803f1b0ead37e89b9a688b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 23:57:44.759158 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:40.493559 7570 trace.go:236] Trace[320261898]: "iptables ChainExists" (30-Oct-2026 23:56:05.320) (total time: 95172ms): Oct 30 23:57:44.759158 ip-10-0-137-81 kubenswrapper[7570]: Trace[320261898]: [1m35.172951978s] [1m35.172951978s] END Oct 30 23:57:44.759158 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.081299 7570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:57:44.759158 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.081334 7570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:57:44.759158 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.081348 7570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:57:44.759158 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.281845 7570 generic.go:356] "Generic (PLEG): container finished" podID="f7c5040c-6dbc-424d-a258-94092e75a027" containerID="d8a0528c523be2bba1198e2108c245b96b5da75b4c7665eb9f0ee23394be4482" exitCode=1 Oct 30 23:57:44.759385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.788854 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" event={"ID":"cb95af9b-7595-4dd9-839a-e698e2a22c07","Type":"ContainerStarted","Data":"7810fe887752c1407d5b0679f1d7482ddd808493153ddc765c93b379508e7ce7"} Oct 30 23:57:44.759385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.790364 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:57:44.759385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.790399 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:57:44.759385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.892950 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wczpt\"" Oct 30 23:57:44.759385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.950384 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-6c5084-openshift-default-dockercfg-bblz7\"" Oct 30 23:57:44.759385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.950531 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v5mxr\"" Oct 30 23:57:44.759385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.950665 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rpncv\"" Oct 30 23:57:44.759385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.951826 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51"} pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 23:57:44.759385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.951885 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" containerID="cri-o://5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51" gracePeriod=30 Oct 30 23:57:44.759636 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.970354 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358"} pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 23:57:44.759636 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.970438 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" containerID="cri-o://e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358" gracePeriod=30 Oct 30 23:57:44.759636 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.973074 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-e15304-openshift-default-dockercfg-qbhbd\"" Oct 30 23:57:44.759636 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.973785 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-23f75e-openshift-default-dockercfg-ngphj\"" Oct 30 23:57:44.759636 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.973894 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ww7vc\"" Oct 30 23:57:44.759636 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.975338 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"default-dockercfg-75htb\"" Oct 30 23:57:44.759636 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.975530 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-200408-dockercfg-zpwrj\"" Oct 30 23:57:44.759636 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.984744 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="metrics-server" containerStatusID={"Type":"cri-o","ID":"d6c071d29384152b592df801487206c5ca8e68ec1e11a4f296680f8452dffa8d"} pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" containerMessage="Container metrics-server failed liveness probe, will be restarted" Oct 30 23:57:44.759876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.984793 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" podUID="ed360b42-1a02-4474-9bf1-2117d1b438f1" containerName="metrics-server" containerID="cri-o://d6c071d29384152b592df801487206c5ca8e68ec1e11a4f296680f8452dffa8d" gracePeriod=170 Oct 30 23:57:44.759876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:42.997425 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b567bb-openshift-default-dockercfg-wvjgm\"" Oct 30 23:57:44.759876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.039853 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xmrvg\"" Oct 30 23:57:44.759876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.040868 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-23f75e-dockercfg-w5fh9\"" Oct 30 23:57:44.759876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.041274 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"default-dockercfg-4fxkj\"" Oct 30 23:57:44.759876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.042023 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-843d8c-dockercfg-ptlb9\"" Oct 30 23:57:44.759876 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.044179 7570 kubelet.go:1648] "Container garbage collection failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Oct 30 23:57:44.759876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.045091 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-82d449-dockercfg-fwwd9\"" Oct 30 23:57:44.759876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.045154 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076"} pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 23:57:44.760095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.045216 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" containerID="cri-o://3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076" gracePeriod=30 Oct 30 23:57:44.760095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.046810 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9"} pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 23:57:44.760095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.046859 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" containerID="cri-o://d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9" gracePeriod=30 Oct 30 23:57:44.760095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.049127 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-86d846-dockercfg-dc664\"" Oct 30 23:57:44.760095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.049380 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-e15304-dockercfg-8mxkq\"" Oct 30 23:57:44.760095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.049666 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-611ef3-dockercfg-m9bvp\"" Oct 30 23:57:44.760095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.049761 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2b4009-openshift-default-dockercfg-mnjpt\"" Oct 30 23:57:44.760095 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.050528 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-djdr2\"" Oct 30 23:57:44.760297 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.051070 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-2q87t\"" Oct 30 23:57:44.760297 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.051110 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f026c1-openshift-default-dockercfg-6hgm8\"" Oct 30 23:57:44.760297 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.051926 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" secret="" err="secret \"payload-processing-e2e-inf-a-82d449-dockercfg-fwwd9\" not found" Oct 30 23:57:44.760297 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.055394 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"7109ce1e7cdee0e506b5064f13410fff49b1fa11c0d3b80c387c43b8850f7c5f"} pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 23:57:44.760297 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.055444 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" containerID="cri-o://7109ce1e7cdee0e506b5064f13410fff49b1fa11c0d3b80c387c43b8850f7c5f" gracePeriod=30 Oct 30 23:57:44.760297 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.055966 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-driver" containerStatusID={"Type":"cri-o","ID":"ef3d34d838ce65d2c862d47a6204e8f9216ca3b7dbfb35b6716b7330c29a9165"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" containerMessage="Container csi-driver failed liveness probe, will be restarted" Oct 30 23:57:44.760512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.056025 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-node-driver-registrar" containerStatusID={"Type":"cri-o","ID":"2a5d66974e4610f8cfba0c638bf3e8679c1cb5b9bb4afc436be4d1c5416ecbdb"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" containerMessage="Container csi-node-driver-registrar failed liveness probe, will be restarted" Oct 30 23:57:44.760512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.056091 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podUID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerName="csi-driver" containerID="cri-o://ef3d34d838ce65d2c862d47a6204e8f9216ca3b7dbfb35b6716b7330c29a9165" gracePeriod=30 Oct 30 23:57:44.760512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.058014 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b"} pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 23:57:44.760512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.058056 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" containerID="cri-o://3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b" gracePeriod=30 Oct 30 23:57:44.760512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.058376 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" secret="" err="secret \"payload-processing-e2e-inf-a-82d449-dockercfg-fwwd9\" not found" Oct 30 23:57:44.760512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.058911 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" secret="" err="secret \"e2e-inf-a-f026c1-openshift-default-dockercfg-6hgm8\" not found" Oct 30 23:57:44.760663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.059617 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" secret="" err="secret \"e2e-inf-a-2b4009-openshift-default-dockercfg-mnjpt\" not found" Oct 30 23:57:44.760663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.061078 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"a122980b7cbc76a173c9f98de07c0e43bb457617d98005f8f366d95243978ad7"} pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 23:57:44.760663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.061120 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" containerID="cri-o://a122980b7cbc76a173c9f98de07c0e43bb457617d98005f8f366d95243978ad7" gracePeriod=30 Oct 30 23:57:44.760663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.064742 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4"} pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 23:57:44.760663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.067053 7570 scope.go:122] "RemoveContainer" containerID="d8a0528c523be2bba1198e2108c245b96b5da75b4c7665eb9f0ee23394be4482" Oct 30 23:57:44.760663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.069280 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" containerID="cri-o://8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4" gracePeriod=30 Oct 30 23:57:44.760663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.077767 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9ef03a-openshift-default-dockercfg-mf4wj\"" Oct 30 23:57:44.760834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.078027 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n9xrj\"" Oct 30 23:57:44.760834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.078253 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5f557\"" Oct 30 23:57:44.760834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.078652 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xj5wh\"" Oct 30 23:57:44.760834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.078720 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qwtvf\"" Oct 30 23:57:44.760834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.078779 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rb9gl\"" Oct 30 23:57:44.760834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.081770 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-32732d-dockercfg-rcltc\"" Oct 30 23:57:44.760834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.082061 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-n7mcg\"" Oct 30 23:57:44.760834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.082251 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-df3b5f-dockercfg-rnjbv\"" Oct 30 23:57:44.760834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.082343 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gskv9\"" Oct 30 23:57:44.760834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.082428 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9tmfn\"" Oct 30 23:57:44.760834 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.082510 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-x7g2z\"" Oct 30 23:57:44.761096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.082659 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-df3b5f-openshift-default-dockercfg-hmtcd\"" Oct 30 23:57:44.761096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.093983 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" secret="" err="secret \"payload-processing-e2e-inf-b-df3b5f-dockercfg-rnjbv\" not found" Oct 30 23:57:44.761096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094303 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ttclb\"" Oct 30 23:57:44.761096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094419 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8gxz9\"" Oct 30 23:57:44.761096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094432 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lzqxs\"" Oct 30 23:57:44.761096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094441 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qbh58\"" Oct 30 23:57:44.761096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094489 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zcrv2\"" Oct 30 23:57:44.761096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094571 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hc9wr\"" Oct 30 23:57:44.761096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094633 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dd3a15-openshift-default-dockercfg-72cjz\"" Oct 30 23:57:44.761096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094647 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"default-dockercfg-l9wjz\"" Oct 30 23:57:44.761096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094716 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-vf4pc\"" Oct 30 23:57:44.761389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094737 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2ccd42-openshift-default-dockercfg-qcc6m\"" Oct 30 23:57:44.761389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094773 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-a3e876-dockercfg-2tm45\"" Oct 30 23:57:44.761389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094836 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a63288-openshift-default-dockercfg-tdk54\"" Oct 30 23:57:44.761389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094884 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4bbvk\"" Oct 30 23:57:44.761389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.094931 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z9rpn\"" Oct 30 23:57:44.761389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.095848 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-82d449-openshift-default-dockercfg-mhmr7\"" Oct 30 23:57:44.761389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.096379 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-82xnp\"" Oct 30 23:57:44.761389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.096866 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"190fd53e5d4cc3e3087e5f2daddbdd346deb07e3206ffa107d2a1d42b517c394"} pod="opendatahub/odh-model-controller-769875db56-4blbf" containerMessage="Container manager failed liveness probe, will be restarted" Oct 30 23:57:44.761389 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.096928 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/odh-model-controller-769875db56-4blbf" podUID="1793bf9b-92db-4dae-bfd6-f86e5b076739" containerName="manager" containerID="cri-o://190fd53e5d4cc3e3087e5f2daddbdd346deb07e3206ffa107d2a1d42b517c394" gracePeriod=10 Oct 30 23:57:44.761641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.097990 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hgj94\"" Oct 30 23:57:44.761641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.097995 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="kube-rbac-proxy-web" containerStatusID={"Type":"cri-o","ID":"e3b93462c6bfe7c40eb3cd7d9d3104119c554709ff072dae0cfec9214ea60d02"} pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" containerMessage="Container kube-rbac-proxy-web failed liveness probe, will be restarted" Oct 30 23:57:44.761641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.098288 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" containerID="cri-o://e3b93462c6bfe7c40eb3cd7d9d3104119c554709ff072dae0cfec9214ea60d02" gracePeriod=120 Oct 30 23:57:44.761641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.098722 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d3dcca-openshift-default-dockercfg-mtkk6\"" Oct 30 23:57:44.761641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.098890 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a63288-dockercfg-5m62n\"" Oct 30 23:57:44.761641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.099423 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fl65j\"" Oct 30 23:57:44.761641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.099592 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1bc837-openshift-default-dockercfg-5htq6\"" Oct 30 23:57:44.761641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.100164 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-885999-dockercfg-tmfpm\"" Oct 30 23:57:44.761641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.100416 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2nz9z\"" Oct 30 23:57:44.761855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.100888 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2mrzn\"" Oct 30 23:57:44.761855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.101506 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-32732d-openshift-default-dockercfg-fvnxc\"" Oct 30 23:57:44.761855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.101601 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lwvlw\"" Oct 30 23:57:44.761855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.102075 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7fc6e9-dockercfg-85m2r\"" Oct 30 23:57:44.761855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.103376 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="cert-manager-webhook" containerStatusID={"Type":"cri-o","ID":"a3d1da19d83c6d20abbe7767b9039dd323558307503a1344bb47b92377be204d"} pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" containerMessage="Container cert-manager-webhook failed liveness probe, will be restarted" Oct 30 23:57:44.761855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.103429 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" containerID="cri-o://a3d1da19d83c6d20abbe7767b9039dd323558307503a1344bb47b92377be204d" gracePeriod=30 Oct 30 23:57:44.761855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.109874 7570 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"c5562a3342bf0ed100c461b7305c75c61de2b03f84d5c91d893824188f07732f"} pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" containerMessage="Container manager failed liveness probe, will be restarted" Oct 30 23:57:44.762038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.109916 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" podUID="d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e" containerName="manager" containerID="cri-o://c5562a3342bf0ed100c461b7305c75c61de2b03f84d5c91d893824188f07732f" gracePeriod=10 Oct 30 23:57:44.762038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.120903 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:57:44.762038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.120935 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:57:44.762038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.120970 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:57:44.762038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121059 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:57:44.762038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121076 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:57:44.762038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121105 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:57:44.762038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121129 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" event={"ID":"f8b86c0c-3f2f-4f9d-be26-948cb9084373","Type":"ContainerStarted","Data":"5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854"} Oct 30 23:57:44.762038 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121186 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" event={"ID":"f8b86c0c-3f2f-4f9d-be26-948cb9084373","Type":"ContainerStarted","Data":"d227395e13ef5d3712a842aa0cba775cedb5e82bed38de62721b43a42c08edeb"} Oct 30 23:57:44.762307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121254 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:57:44.762307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121271 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:57:44.762307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121358 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:57:44.762307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121377 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:57:44.762307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121400 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" event={"ID":"f7c5040c-6dbc-424d-a258-94092e75a027","Type":"ContainerDied","Data":"d8a0528c523be2bba1198e2108c245b96b5da75b4c7665eb9f0ee23394be4482"} Oct 30 23:57:44.762307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121442 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:57:44.762307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121536 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:57:44.762307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121560 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:57:44.762307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121581 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:57:44.762307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121612 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:57:44.762635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121641 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:57:44.762635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121709 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:57:44.762635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121725 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:57:44.762635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121754 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:57:44.762635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121825 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:57:44.762635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121847 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" Oct 30 23:57:44.762635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121881 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:57:44.762635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121923 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:57:44.762635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.121993 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:57:44.762635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.122009 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:57:44.762903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.123477 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": dial tcp 10.134.0.135:8443: connect: connection refused" Oct 30 23:57:44.762903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.124517 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:57:44.762903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.124836 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:57:44.762903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.125119 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:57:44.762903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.125428 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:57:44.762903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.125723 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:57:44.762903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.125816 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": dial tcp 10.134.0.157:8443: connect: connection refused" Oct 30 23:57:44.762903 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.125895 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-4blbf" podUID="1793bf9b-92db-4dae-bfd6-f86e5b076739" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/readyz\": dial tcp 10.134.0.25:8081: connect: connection refused" Oct 30 23:57:44.763100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.126060 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:57:44.763100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.126294 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:57:44.763100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.126370 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:57:44.763100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.126573 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:57:44.763100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.126584 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused" Oct 30 23:57:44.763100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.126683 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:57:44.763100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.126848 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused" Oct 30 23:57:44.763100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.126949 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:57:44.763100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.127187 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:57:44.763346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.127661 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:57:44.763346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.127852 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:57:44.763346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.128044 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:57:44.763346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.128344 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:57:44.763346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.132572 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:57:44.763346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.132798 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-89shx" Oct 30 23:57:44.763346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.132990 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:57:44.763346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.133106 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:57:44.763346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.133320 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:57:44.763346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.133795 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:57:44.763346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.133966 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:57:44.763346 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.134069 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h4hmx" Oct 30 23:57:44.763674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.134151 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:57:44.763674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.134881 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:57:44.763674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.135107 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:57:44.763674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.135245 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:57:44.763674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.135531 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:57:44.763674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.135685 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:57:44.763674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.136043 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:57:44.763674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.136271 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": dial tcp 10.134.0.142:8443: connect: connection refused" Oct 30 23:57:44.763674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.136344 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:57:44.763898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.137375 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": dial tcp 10.134.0.47:8443: connect: connection refused" Oct 30 23:57:44.763898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.137618 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": dial tcp 10.134.0.131:8443: connect: connection refused" Oct 30 23:57:44.763898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.140118 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/healthz\": dial tcp 10.134.0.23:8081: connect: connection refused" Oct 30 23:57:44.763898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.140167 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:57:44.763898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.140292 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:57:44.763898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.140437 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:57:44.763898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.140745 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:57:44.763898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.140894 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:57:44.764093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.140890 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": dial tcp 10.134.0.142:8443: connect: connection refused" Oct 30 23:57:44.764093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.141034 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:57:44.764093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.141232 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:57:44.764093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.141755 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 30 23:57:44.764093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.141954 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 30 23:57:44.764093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.142076 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:57:44.764093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.142297 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:57:44.764093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.144540 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 30 23:57:44.764093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.144705 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:57:44.764093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.146019 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:57:44.764339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.146736 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:57:44.764339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.146927 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:57:44.764339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.147161 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:57:44.764339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.147259 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:57:44.764339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.149509 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 23:57:44.764339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.149738 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.764339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.149741 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:57:44.764339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.149798 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:57:44.764339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.149828 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:57:44.764339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.149834 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:57:44.764339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150133 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:57:44.764630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150175 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:57:44.764630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150199 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:57:44.764630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150223 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 30 23:57:44.764630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150247 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 30 23:57:44.764630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150747 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 30 23:57:44.764630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150792 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:57:44.764630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150825 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:57:44.764630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150858 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:57:44.764630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150891 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:57:44.764630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150923 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 30 23:57:44.764630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150948 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:57:44.764880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.150980 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:57:44.764880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151012 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:57:44.764880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151045 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:57:44.764880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151080 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:57:44.764880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151115 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:57:44.764880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151145 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:57:44.764880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151179 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:57:44.764880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151212 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:57:44.764880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151243 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:57:44.764880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151287 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:57:44.764880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151338 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:57:44.765145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151369 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:57:44.765145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151403 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:57:44.765145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151432 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:57:44.765145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151602 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:57:44.765145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151638 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:57:44.765145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151723 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 30 23:57:44.765145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151758 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:57:44.765145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151795 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:57:44.765145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151831 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 30 23:57:44.765145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151876 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:57:44.765145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151917 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:57:44.765412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151950 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:57:44.765412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.151985 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 30 23:57:44.765412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.152015 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:57:44.765412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.152059 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:57:44.765412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.152190 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:57:44.765412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.152323 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:57:44.765412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.154359 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 30 23:57:44.765412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.153167 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.765412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.164995 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" Oct 30 23:57:44.765412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.167915 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" Oct 30 23:57:44.765412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.168136 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" Oct 30 23:57:44.765696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.168643 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:57:44.765696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.169504 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:57:44.765696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.170107 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:57:44.765696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.170500 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-89shx" Oct 30 23:57:44.765696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.170631 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:57:44.765696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.171706 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:57:44.765696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.171763 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:57:44.765696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.171810 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:57:44.765696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.171860 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h4hmx" Oct 30 23:57:44.765696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.171904 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:57:44.765696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.171945 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:57:44.765696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.171994 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:57:44.765980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172036 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 30 23:57:44.765980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172079 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 30 23:57:44.765980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172120 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 30 23:57:44.765980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172160 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 30 23:57:44.765980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172203 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 30 23:57:44.765980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172249 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:57:44.765980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172289 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:57:44.765980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172327 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:57:44.765980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172366 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:57:44.765980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172436 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 30 23:57:44.765980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172489 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 30 23:57:44.766287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172531 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:57:44.766287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172573 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:57:44.766287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172613 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:57:44.766287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172655 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:57:44.766287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172695 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 30 23:57:44.766287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172738 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:57:44.766287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172778 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:57:44.766287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172819 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 30 23:57:44.766287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172858 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:57:44.766287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172896 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 30 23:57:44.766287 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172936 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 30 23:57:44.766576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.172976 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:57:44.766576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.173015 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 30 23:57:44.766576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.173054 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 30 23:57:44.766576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.173094 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 30 23:57:44.766576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.173131 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 30 23:57:44.766576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.173170 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 30 23:57:44.766576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.173210 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" Oct 30 23:57:44.766576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.173250 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:57:44.766576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.173290 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 30 23:57:44.766576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.173331 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:57:44.766576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.176733 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:57:44.766842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.183487 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": dial tcp 10.134.0.151:8443: connect: connection refused" Oct 30 23:57:44.766842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.183606 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": dial tcp 10.134.0.143:8443: connect: connection refused" Oct 30 23:57:44.766842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.185520 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:57:44.766842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.185626 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" Oct 30 23:57:44.766842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.186754 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:57:44.766842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.186828 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:57:44.766842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.191197 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:57:44.766842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.191809 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.766842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.191887 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 30 23:57:44.767059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.192027 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-885999-serving-cert\"" Oct 30 23:57:44.767059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.192095 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.767059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.192233 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:57:44.767059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.193398 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 30 23:57:44.767059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.212864 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:57:44.767059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.212956 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:57:44.767059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.214196 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:57:44.767059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.214874 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-611ef3\"" Oct 30 23:57:44.767059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.216599 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:57:44.767059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.217607 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:57:44.767059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.220258 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.767363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.223009 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:57:44.767363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.223061 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2b4009\"" Oct 30 23:57:44.767363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.223166 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 23:57:44.767363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.224423 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-df3b5f\"" Oct 30 23:57:44.767363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.224857 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 30 23:57:44.767363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.224900 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:57:44.767363 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.227662 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-2b4009: configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:57:44.767363 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.227976 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume podName:b12dc177-11a1-40b0-9fdf-75993ab176e6 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:43.727957249 +0000 UTC m=+2275.246958320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume") pod "payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" (UID: "b12dc177-11a1-40b0-9fdf-75993ab176e6") : configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:57:44.767704 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.230757 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-df3b5f: configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:57:44.767704 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.230812 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume podName:e9c9e5ed-908c-463d-8f4f-395a80ababdd nodeName:}" failed. No retries permitted until 2026-10-30 23:57:43.730796646 +0000 UTC m=+2275.249797713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" (UID: "e9c9e5ed-908c-463d-8f4f-395a80ababdd") : configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:57:44.767704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.244931 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:57:44.767704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.245076 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-23f75e\"" Oct 30 23:57:44.767704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.250679 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:57:44.767704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.258221 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.767704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.258785 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.767704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.258790 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:57:44.767704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.258834 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 23:57:44.769455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.258880 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-6c5084-serving-cert\"" Oct 30 23:57:44.769455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.258904 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:57:44.769455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.259451 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7b0299\"" Oct 30 23:57:44.769455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.259491 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 23:57:44.769455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.259796 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 23:57:44.769455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.259837 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-843d8c\"" Oct 30 23:57:44.769455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.259878 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a63288\"" Oct 30 23:57:44.769455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.259902 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:57:44.769455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.259934 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 23:57:44.769455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.259964 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:57:44.769455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.260512 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.769455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.261111 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 23:57:44.770005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.261710 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"premium-w4-843d8c-kserve-self-signed-certs\"" Oct 30 23:57:44.770005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.261754 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7fc6e9-serving-cert\"" Oct 30 23:57:44.770005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.261837 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 23:57:44.770005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.261884 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 23:57:44.770005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.262129 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.770005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.262368 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-82d449\"" Oct 30 23:57:44.770005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.262400 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 23:57:44.770005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.262433 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 23:57:44.770005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.262642 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 23:57:44.770005 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.262924 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 23:57:44.770005 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.263982 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:44.770350 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.264042 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume podName:aa08b804-5246-4fb8-a68c-cf7af9ba4a54 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:43.764023379 +0000 UTC m=+2275.283024449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume") pod "payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" (UID: "aa08b804-5246-4fb8-a68c-cf7af9ba4a54") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:44.770350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.265410 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 23:57:44.770350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.274429 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.770350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.274687 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.770350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.276349 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:57:44.770350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.285362 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 23:57:44.770350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.285384 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.770350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.345431 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 23:57:44.770350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.345531 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:57:44.770687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.345527 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.770687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.345610 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 23:57:44.770687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.345720 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1j1fi1cun0mic\"" Oct 30 23:57:44.770687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.350339 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7fh9lroec4t9q\"" Oct 30 23:57:44.770687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.350434 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:57:44.770687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.350575 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 23:57:44.770687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.350638 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 23:57:44.770687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.355804 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.770687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.355826 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 23:57:44.770687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.356385 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-a3e876\"" Oct 30 23:57:44.770687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.356452 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 23:57:44.770687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.356558 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 23:57:44.773491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.360672 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.773491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.360923 7570 generic.go:356] "Generic (PLEG): container finished" podID="ee540c2e-f028-44ee-be53-d178b286fb02" containerID="318dfc9987ebf2f548bc401821e12fd61ce5837c82803f1b0ead37e89b9a688b" exitCode=1 Oct 30 23:57:44.773491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.360953 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-p5xwl" event={"ID":"ee540c2e-f028-44ee-be53-d178b286fb02","Type":"ContainerDied","Data":"318dfc9987ebf2f548bc401821e12fd61ce5837c82803f1b0ead37e89b9a688b"} Oct 30 23:57:44.773491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.361349 7570 scope.go:122] "RemoveContainer" containerID="318dfc9987ebf2f548bc401821e12fd61ce5837c82803f1b0ead37e89b9a688b" Oct 30 23:57:44.773491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.362098 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.773491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.363129 7570 generic.go:356] "Generic (PLEG): container finished" podID="2b47b3cc-9735-45f1-8d87-7edd6ef95dd8" containerID="e744dd43c6e82155215305ded5379603f8103350e6f2fb7975b8e476b2e81b76" exitCode=0 Oct 30 23:57:44.773491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.363192 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" event={"ID":"2b47b3cc-9735-45f1-8d87-7edd6ef95dd8","Type":"ContainerDied","Data":"e744dd43c6e82155215305ded5379603f8103350e6f2fb7975b8e476b2e81b76"} Oct 30 23:57:44.773491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.363593 7570 scope.go:122] "RemoveContainer" containerID="e744dd43c6e82155215305ded5379603f8103350e6f2fb7975b8e476b2e81b76" Oct 30 23:57:44.773491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.364077 7570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:57:44.773491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.364633 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 23:57:44.773847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.364732 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 23:57:44.773847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.365365 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 23:57:44.773847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.365849 7570 generic.go:356] "Generic (PLEG): container finished" podID="347b9a5b-729f-419a-bce5-749ac4aca5b5" containerID="227922d5b20c1762b1662970a8dc5aa42f92fd5e306aca4c173a37e02e3feb31" exitCode=1 Oct 30 23:57:44.773847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.365939 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" event={"ID":"347b9a5b-729f-419a-bce5-749ac4aca5b5","Type":"ContainerDied","Data":"227922d5b20c1762b1662970a8dc5aa42f92fd5e306aca4c173a37e02e3feb31"} Oct 30 23:57:44.773847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.366394 7570 scope.go:122] "RemoveContainer" containerID="227922d5b20c1762b1662970a8dc5aa42f92fd5e306aca4c173a37e02e3feb31" Oct 30 23:57:44.773847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.368132 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 23:57:44.773847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.368435 7570 generic.go:356] "Generic (PLEG): container finished" podID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerID="086b42ff31df3806fabe032ac13c48753bc1001a7c213cbaa603d1d1f2bdd8fb" exitCode=1 Oct 30 23:57:44.773847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.368490 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" event={"ID":"20fc3e5f-362e-41f7-bb52-2b171d7af818","Type":"ContainerDied","Data":"086b42ff31df3806fabe032ac13c48753bc1001a7c213cbaa603d1d1f2bdd8fb"} Oct 30 23:57:44.773847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.368881 7570 scope.go:122] "RemoveContainer" containerID="086b42ff31df3806fabe032ac13c48753bc1001a7c213cbaa603d1d1f2bdd8fb" Oct 30 23:57:44.773847 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.370115 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 23:57:44.774184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.370477 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.774184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.370632 7570 generic.go:356] "Generic (PLEG): container finished" podID="1793bf9b-92db-4dae-bfd6-f86e5b076739" containerID="190fd53e5d4cc3e3087e5f2daddbdd346deb07e3206ffa107d2a1d42b517c394" exitCode=1 Oct 30 23:57:44.774184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.371726 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-4blbf" event={"ID":"1793bf9b-92db-4dae-bfd6-f86e5b076739","Type":"ContainerDied","Data":"190fd53e5d4cc3e3087e5f2daddbdd346deb07e3206ffa107d2a1d42b517c394"} Oct 30 23:57:44.774184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.372295 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" secret="" err="secret \"payload-processing-e2e-inf-b-df3b5f-dockercfg-rnjbv\" not found" Oct 30 23:57:44.774184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.376067 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" secret="" err="secret \"e2e-inf-a-2b4009-openshift-default-dockercfg-mnjpt\" not found" Oct 30 23:57:44.774184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.377970 7570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:57:44.774184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.379009 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:57:44.774184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.379605 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:57:44.774184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.383425 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" secret="" err="secret \"payload-processing-e2e-inf-a-82d449-dockercfg-fwwd9\" not found" Oct 30 23:57:44.774642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.387143 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"distinct-w4-843d8c-kserve-self-signed-certs\"" Oct 30 23:57:44.774642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.387200 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-9ef03a-serving-cert\"" Oct 30 23:57:44.774642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.388188 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 23:57:44.774642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.388424 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" secret="" err="secret \"payload-processing-e2e-inf-a-82d449-dockercfg-fwwd9\" not found" Oct 30 23:57:44.774642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.388897 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 23:57:44.774642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.389766 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-885999\"" Oct 30 23:57:44.774642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.390544 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-200408\"" Oct 30 23:57:44.774642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.391652 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-a3e876-serving-cert\"" Oct 30 23:57:44.774642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.392325 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:57:44.774642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.392670 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-843d8c-serving-cert\"" Oct 30 23:57:44.774642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.393007 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 23:57:44.775059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.393164 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"distinct-2-w4-843d8c-kserve-self-signed-certs\"" Oct 30 23:57:44.775059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.395640 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 23:57:44.775059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.395878 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 23:57:44.775059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.395971 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" secret="" err="secret \"e2e-inf-a-f026c1-openshift-default-dockercfg-6hgm8\" not found" Oct 30 23:57:44.775059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.396023 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 23:57:44.775059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.396078 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 23:57:44.775059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.396431 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-86d846\"" Oct 30 23:57:44.775059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.400106 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 23:57:44.775059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.400149 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 23:57:44.775059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.400199 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.775059 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.406274 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 23:57:44.775502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.406756 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 23:57:44.775502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.406856 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.775502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.406983 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 23:57:44.775502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.407019 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-e15304\"" Oct 30 23:57:44.775502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.407091 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.775502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.407197 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 23:57:44.775502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.407213 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 23:57:44.775502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.407329 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 23:57:44.775502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.407360 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 23:57:44.775502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.407480 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.775502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.407597 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"simulator-w2-9ef03a-kserve-self-signed-certs\"" Oct 30 23:57:44.775502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.407745 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.775913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.407760 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 23:57:44.775913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.407814 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1bc837-serving-cert\"" Oct 30 23:57:44.775913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.407846 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.775913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.408160 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 23:57:44.775913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.408826 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 23:57:44.775913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.408896 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 23:57:44.775913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.408945 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"simulator-w0-d3dcca-kserve-self-signed-certs\"" Oct 30 23:57:44.775913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.408960 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7fc6e9\"" Oct 30 23:57:44.775913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.415637 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 23:57:44.775913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.415849 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 23:57:44.775913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.421580 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.775913 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.421597 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.776349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.422715 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 23:57:44.776349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.422831 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 23:57:44.776349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.422973 7570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="cert-manager/cert-manager-597b95cb84-p5xwl" Oct 30 23:57:44.776349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.422983 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.776349 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.451851 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:44.776349 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.451908 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume podName:b4e6db40-bf7c-4440-bed0-8ec6e70cab0f nodeName:}" failed. No retries permitted until 2026-10-30 23:57:43.951890223 +0000 UTC m=+2275.470891298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" (UID: "b4e6db40-bf7c-4440-bed0-8ec6e70cab0f") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:44.776349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.453187 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:57:44.776349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.453958 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 23:57:44.776696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.454515 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:57:44.776696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.454560 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 23:57:44.776696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.454801 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 23:57:44.776696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.457069 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-32732d\"" Oct 30 23:57:44.776696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.457174 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3dcca\"/\"distinct-2-w0-d3dcca-kserve-self-signed-certs\"" Oct 30 23:57:44.776696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.457558 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-anck8g3riltac\"" Oct 30 23:57:44.776696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.461692 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:57:44.776696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.462170 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:57:44.776696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.462552 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 23:57:44.776696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.462575 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:57:44.776696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.463884 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"kube-root-ca.crt\"" Oct 30 23:57:44.776696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.465024 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 23:57:44.777145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.474865 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"distinct-2-w6-e15304-kserve-self-signed-certs\"" Oct 30 23:57:44.777145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.479882 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:57:44.777145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.479890 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"openshift-service-ca.crt\"" Oct 30 23:57:44.777145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.662136 7570 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 23:57:44.777145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.703498 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": dial tcp 10.134.0.13:9091: connect: connection refused" start-of-body= Oct 30 23:57:44.777145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:43.703544 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": dial tcp 10.134.0.13:9091: connect: connection refused" Oct 30 23:57:44.777145 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.775635 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:44.777387 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.775686 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume podName:aa08b804-5246-4fb8-a68c-cf7af9ba4a54 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:44.775670801 +0000 UTC m=+2276.294671893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume") pod "payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" (UID: "aa08b804-5246-4fb8-a68c-cf7af9ba4a54") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:44.777387 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.775709 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-2b4009: configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:57:44.777387 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.775750 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume podName:b12dc177-11a1-40b0-9fdf-75993ab176e6 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:44.775738975 +0000 UTC m=+2276.294740047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume") pod "payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" (UID: "b12dc177-11a1-40b0-9fdf-75993ab176e6") : configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:57:44.777387 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.775863 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-df3b5f: configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:57:44.777536 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.775893 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume podName:e9c9e5ed-908c-463d-8f4f-395a80ababdd nodeName:}" failed. No retries permitted until 2026-10-30 23:57:44.775886089 +0000 UTC m=+2276.294887192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" (UID: "e9c9e5ed-908c-463d-8f4f-395a80ababdd") : configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:57:44.777536 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.980138 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:44.777536 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:43.980217 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume podName:b4e6db40-bf7c-4440-bed0-8ec6e70cab0f nodeName:}" failed. No retries permitted until 2026-10-30 23:57:44.980197244 +0000 UTC m=+2276.499198337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" (UID: "b4e6db40-bf7c-4440-bed0-8ec6e70cab0f") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:44.777536 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:44.175593 7570 patch_prober.go:29] interesting pod/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:44.777653 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:44.175641 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:44.777653 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:44.338104 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:57:44.777653 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:44.385665 7570 generic.go:356] "Generic (PLEG): container finished" podID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerID="8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4" exitCode=0 Oct 30 23:57:44.777653 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:44.385750 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" event={"ID":"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5","Type":"ContainerDied","Data":"8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4"} Oct 30 23:57:44.777653 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:44.389155 7570 generic.go:356] "Generic (PLEG): container finished" podID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerID="e3b93462c6bfe7c40eb3cd7d9d3104119c554709ff072dae0cfec9214ea60d02" exitCode=0 Oct 30 23:57:44.777653 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:44.389237 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" event={"ID":"d189925d-3211-43c6-8bf0-e9aca119e00e","Type":"ContainerDied","Data":"e3b93462c6bfe7c40eb3cd7d9d3104119c554709ff072dae0cfec9214ea60d02"} Oct 30 23:57:44.777653 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:44.390978 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" secret="" err="secret \"e2e-inf-a-f026c1-openshift-default-dockercfg-6hgm8\" not found" Oct 30 23:57:44.777843 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:44.397332 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" secret="" err="secret \"e2e-inf-a-2b4009-openshift-default-dockercfg-mnjpt\" not found" Oct 30 23:57:44.777843 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:44.397791 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" secret="" err="secret \"payload-processing-e2e-inf-a-82d449-dockercfg-fwwd9\" not found" Oct 30 23:57:44.777843 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:44.397801 7570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" secret="" err="secret \"payload-processing-e2e-inf-a-82d449-dockercfg-fwwd9\" not found" Oct 30 23:57:44.777843 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:44.500066 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:57:44.812853 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:44.812548 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-2b4009: configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:57:44.812853 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:44.812732 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume podName:b12dc177-11a1-40b0-9fdf-75993ab176e6 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:46.812712783 +0000 UTC m=+2278.331713850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume") pod "payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" (UID: "b12dc177-11a1-40b0-9fdf-75993ab176e6") : configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:57:44.812853 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:44.812760 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-df3b5f: configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:57:44.812853 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:44.812822 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume podName:e9c9e5ed-908c-463d-8f4f-395a80ababdd nodeName:}" failed. No retries permitted until 2026-10-30 23:57:46.812806997 +0000 UTC m=+2278.331808089 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" (UID: "e9c9e5ed-908c-463d-8f4f-395a80ababdd") : configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:57:44.815028 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:44.814827 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:44.815028 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:44.814938 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume podName:aa08b804-5246-4fb8-a68c-cf7af9ba4a54 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:46.814908959 +0000 UTC m=+2278.333910046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume") pod "payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" (UID: "aa08b804-5246-4fb8-a68c-cf7af9ba4a54") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:44.902977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:44.901206 7570 plugin_watcher.go:214] "Removing socket path from desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:57:45.021303 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:45.020934 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:45.021303 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:45.021041 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume podName:b4e6db40-bf7c-4440-bed0-8ec6e70cab0f nodeName:}" failed. No retries permitted until 2026-10-30 23:57:47.02101781 +0000 UTC m=+2278.540018877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" (UID: "b4e6db40-bf7c-4440-bed0-8ec6e70cab0f") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:45.037983 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.037867 7570 trace.go:236] Trace[481245271]: "Calculate volume metrics of manifests for pod opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" (30-Oct-2026 23:55:05.946) (total time: 159091ms): Oct 30 23:57:45.037983 ip-10-0-137-81 kubenswrapper[7570]: Trace[481245271]: [2m39.091510389s] [2m39.091510389s] END Oct 30 23:57:45.084958 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.084873 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": dial tcp 10.134.0.151:8443: connect: connection refused" Oct 30 23:57:45.085325 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.085134 7570 csi_plugin.go:271] kubernetes.io/csi: registrationHandler.DeRegisterPlugin request for plugin ebs.csi.aws.com, endpoint /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:57:45.108709 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.108672 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" podUID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerName="csi-node-driver-registrar" containerID="cri-o://2a5d66974e4610f8cfba0c638bf3e8679c1cb5b9bb4afc436be4d1c5416ecbdb" gracePeriod=30 Oct 30 23:57:45.142935 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.142907 7570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:57:45.152761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.152729 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2","openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx","openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr","openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght","openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g","openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c"] Oct 30 23:57:45.154145 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.084400 7570 reconciler.go:150] "OperationExecutor.UnregisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"0001-01-01T00:00:00Z","UUID":"a58e340b-7260-4c3d-97cd-a707ebe7a46b","Handler":{},"Name":"ebs.csi.aws.com","Endpoint":"/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock"} Oct 30 23:57:45.217648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.217601 7570 patch_prober.go:29] interesting pod/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:57:45.217787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.217710 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:57:45.256590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.256546 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": dial tcp 10.134.0.143:8443: connect: connection refused" Oct 30 23:57:45.532531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.532506 7570 trace.go:236] Trace[1893387156]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" (30-Oct-2026 23:55:39.493) (total time: 126038ms): Oct 30 23:57:45.532531 ip-10-0-137-81 kubenswrapper[7570]: Trace[1893387156]: [2m6.038688203s] [2m6.038688203s] END Oct 30 23:57:45.613669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.613519 7570 generic.go:356] "Generic (PLEG): container finished" podID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerID="a3d1da19d83c6d20abbe7767b9039dd323558307503a1344bb47b92377be204d" exitCode=0 Oct 30 23:57:45.613669 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.613560 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" event={"ID":"baaa353f-43a5-43e5-a73c-92d1840ebdae","Type":"ContainerDied","Data":"a3d1da19d83c6d20abbe7767b9039dd323558307503a1344bb47b92377be204d"} Oct 30 23:57:45.707112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.706765 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log" Oct 30 23:57:45.707112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.706817 7570 generic.go:356] "Generic (PLEG): container finished" podID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerID="ef3d34d838ce65d2c862d47a6204e8f9216ca3b7dbfb35b6716b7330c29a9165" exitCode=2 Oct 30 23:57:45.707112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.706953 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" event={"ID":"29c3d1e4-09b5-4712-ad6d-9a32434def2d","Type":"ContainerDied","Data":"ef3d34d838ce65d2c862d47a6204e8f9216ca3b7dbfb35b6716b7330c29a9165"} Oct 30 23:57:45.711799 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.711590 7570 generic.go:356] "Generic (PLEG): container finished" podID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerID="3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076" exitCode=0 Oct 30 23:57:45.711799 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.711749 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" event={"ID":"2533ebe5-857f-4255-8c85-5a526fe1062b","Type":"ContainerDied","Data":"3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076"} Oct 30 23:57:45.714425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.714224 7570 generic.go:356] "Generic (PLEG): container finished" podID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerID="3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b" exitCode=0 Oct 30 23:57:45.714425 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.714344 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" event={"ID":"c15181ee-3e9a-40c9-a029-9ee504a971fa","Type":"ContainerDied","Data":"3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b"} Oct 30 23:57:45.717277 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.717257 7570 generic.go:356] "Generic (PLEG): container finished" podID="4af4419c-fffd-463e-9577-94bad299abc6" containerID="7109ce1e7cdee0e506b5064f13410fff49b1fa11c0d3b80c387c43b8850f7c5f" exitCode=0 Oct 30 23:57:45.717485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.717357 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" event={"ID":"4af4419c-fffd-463e-9577-94bad299abc6","Type":"ContainerDied","Data":"7109ce1e7cdee0e506b5064f13410fff49b1fa11c0d3b80c387c43b8850f7c5f"} Oct 30 23:57:45.719662 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.719641 7570 generic.go:356] "Generic (PLEG): container finished" podID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerID="a122980b7cbc76a173c9f98de07c0e43bb457617d98005f8f366d95243978ad7" exitCode=0 Oct 30 23:57:45.720179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.719763 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" event={"ID":"6b3d259c-4f6c-4033-b6dd-1059616fe3b4","Type":"ContainerDied","Data":"a122980b7cbc76a173c9f98de07c0e43bb457617d98005f8f366d95243978ad7"} Oct 30 23:57:45.720502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.720354 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" containerID="cri-o://0bde901d48fa5140c259012bee5dd70f0ef26c4ba298b70a93e1e9c3ff2d7acd" gracePeriod=30 Oct 30 23:57:45.720885 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:45.720827 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" podUID="f68436c1-d900-4063-b853-cb9be78ceae6" containerName="istio-proxy" containerID="cri-o://c30617db6774365f2605f645b02abd554fc4a67999922b27bc2763f1ad73c11a" gracePeriod=30 Oct 30 23:57:46.891591 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:46.891553 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:46.892105 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:46.891645 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume podName:aa08b804-5246-4fb8-a68c-cf7af9ba4a54 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:50.891624663 +0000 UTC m=+2282.410625733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume") pod "payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" (UID: "aa08b804-5246-4fb8-a68c-cf7af9ba4a54") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:46.892105 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:46.891833 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-2b4009: configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:57:46.892105 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:46.891908 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume podName:b12dc177-11a1-40b0-9fdf-75993ab176e6 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:50.891888648 +0000 UTC m=+2282.410889736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume") pod "payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" (UID: "b12dc177-11a1-40b0-9fdf-75993ab176e6") : configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:57:46.892105 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:46.891982 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-df3b5f: configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:57:46.892267 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:46.892070 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume podName:e9c9e5ed-908c-463d-8f4f-395a80ababdd nodeName:}" failed. No retries permitted until 2026-10-30 23:57:50.892052131 +0000 UTC m=+2282.411053218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" (UID: "e9c9e5ed-908c-463d-8f4f-395a80ababdd") : configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:57:47.078219 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:47.078099 7570 generic.go:356] "Generic (PLEG): container finished" podID="d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e" containerID="c5562a3342bf0ed100c461b7305c75c61de2b03f84d5c91d893824188f07732f" exitCode=0 Oct 30 23:57:47.078696 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:47.078662 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" event={"ID":"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e","Type":"ContainerDied","Data":"c5562a3342bf0ed100c461b7305c75c61de2b03f84d5c91d893824188f07732f"} Oct 30 23:57:47.082552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:47.082531 7570 generic.go:356] "Generic (PLEG): container finished" podID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerID="e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358" exitCode=0 Oct 30 23:57:47.082743 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:47.082586 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" event={"ID":"eedd4bdb-1495-47e6-884d-5ce26a05e72b","Type":"ContainerDied","Data":"e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358"} Oct 30 23:57:47.098090 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:47.098057 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:47.098252 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:47.098140 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume podName:b4e6db40-bf7c-4440-bed0-8ec6e70cab0f nodeName:}" failed. No retries permitted until 2026-10-30 23:57:51.098120782 +0000 UTC m=+2282.617121862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" (UID: "b4e6db40-bf7c-4440-bed0-8ec6e70cab0f") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:47.450403 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:47.450326 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:57:48.122990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.122939 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": dial tcp 10.134.0.157:8443: connect: connection refused" Oct 30 23:57:48.123606 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.123005 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": dial tcp 10.134.0.135:8443: connect: connection refused" Oct 30 23:57:48.123714 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.123657 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": dial tcp 10.134.0.20:6080: connect: connection refused" Oct 30 23:57:48.126955 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.126929 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused" Oct 30 23:57:48.136182 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.136151 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": dial tcp 10.134.0.131:8443: connect: connection refused" Oct 30 23:57:48.136751 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.136705 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": dial tcp 10.134.0.47:8443: connect: connection refused" Oct 30 23:57:48.142026 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.141997 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": dial tcp 10.134.0.142:8443: connect: connection refused" Oct 30 23:57:48.151155 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.151132 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log" Oct 30 23:57:48.151356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.151335 7570 generic.go:356] "Generic (PLEG): container finished" podID="29c3d1e4-09b5-4712-ad6d-9a32434def2d" containerID="2a5d66974e4610f8cfba0c638bf3e8679c1cb5b9bb4afc436be4d1c5416ecbdb" exitCode=0 Oct 30 23:57:48.151609 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.151404 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" event={"ID":"29c3d1e4-09b5-4712-ad6d-9a32434def2d","Type":"ContainerDied","Data":"2a5d66974e4610f8cfba0c638bf3e8679c1cb5b9bb4afc436be4d1c5416ecbdb"} Oct 30 23:57:48.154319 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.154298 7570 generic.go:356] "Generic (PLEG): container finished" podID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerID="5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51" exitCode=0 Oct 30 23:57:48.154572 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.154556 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" event={"ID":"06e6c870-08ba-4b3c-a26d-45397d5ca730","Type":"ContainerDied","Data":"5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51"} Oct 30 23:57:48.157130 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.157107 7570 generic.go:356] "Generic (PLEG): container finished" podID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerID="d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9" exitCode=0 Oct 30 23:57:48.157235 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.157225 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" event={"ID":"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903","Type":"ContainerDied","Data":"d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9"} Oct 30 23:57:48.703715 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.703683 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": dial tcp 10.134.0.13:9091: connect: connection refused" start-of-body= Oct 30 23:57:48.704037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:48.704003 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": dial tcp 10.134.0.13:9091: connect: connection refused" Oct 30 23:57:50.072140 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:50.072035 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": dial tcp 10.134.0.151:8443: connect: connection refused" Oct 30 23:57:50.072697 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:50.072203 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:57:50.153478 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:50.153430 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" containerID="cri-o://437b364aa15062825ed4d26c6d17ec8b15ba4dd0916980db86c3f0f71ca026e9" gracePeriod=25 Oct 30 23:57:50.197966 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:50.197935 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" event={"ID":"20fc3e5f-362e-41f7-bb52-2b171d7af818","Type":"ContainerStarted","Data":"0f67fc61e43c08e290895a1470a3abdce4b051f8670c616c91d04c0a03dc331d"} Oct 30 23:57:50.255928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:50.255887 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": dial tcp 10.134.0.143:8443: connect: connection refused" Oct 30 23:57:50.256094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:50.256019 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:57:50.720906 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:50.720869 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" podUID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" containerName="payload-pre-processing" containerID="cri-o://0170737f67453b691218a3069bf111fefb4d960cddcbbb0718f56f6f8b6dd3b2" gracePeriod=25 Oct 30 23:57:50.720906 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:50.720898 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" podUID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" containerName="payload-processing" containerID="cri-o://4c9d727f8c05504972593ae1b6240121490aba70db54418fa228de5c0ea97248" gracePeriod=25 Oct 30 23:57:50.721099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:50.720880 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" podUID="b12dc177-11a1-40b0-9fdf-75993ab176e6" containerName="payload-processing" containerID="cri-o://a2397d1a972d4c1e20267b88f3965c58ed65da73260e8e63c61ae66c513cbdab" gracePeriod=25 Oct 30 23:57:50.962735 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:50.962703 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:50.962906 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:50.962789 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume podName:aa08b804-5246-4fb8-a68c-cf7af9ba4a54 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:58.962768192 +0000 UTC m=+2290.481769263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume") pod "payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" (UID: "aa08b804-5246-4fb8-a68c-cf7af9ba4a54") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:50.962976 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:50.962958 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-2b4009: configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:57:50.963096 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:50.963084 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume podName:b12dc177-11a1-40b0-9fdf-75993ab176e6 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:58.963068014 +0000 UTC m=+2290.482069150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume") pod "payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" (UID: "b12dc177-11a1-40b0-9fdf-75993ab176e6") : configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:57:50.963160 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:50.963021 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-df3b5f: configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:57:50.963160 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:50.963123 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume podName:e9c9e5ed-908c-463d-8f4f-395a80ababdd nodeName:}" failed. No retries permitted until 2026-10-30 23:57:58.963115755 +0000 UTC m=+2290.482116823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" (UID: "e9c9e5ed-908c-463d-8f4f-395a80ababdd") : configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:57:51.167042 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:51.167011 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:51.167530 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:57:51.167096 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume podName:b4e6db40-bf7c-4440-bed0-8ec6e70cab0f nodeName:}" failed. No retries permitted until 2026-10-30 23:57:59.167076612 +0000 UTC m=+2290.686077695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" (UID: "b4e6db40-bf7c-4440-bed0-8ec6e70cab0f") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:57:53.121676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:53.121634 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": dial tcp 10.134.0.135:8443: connect: connection refused" Oct 30 23:57:53.122270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:53.122006 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-4blbf" podUID="1793bf9b-92db-4dae-bfd6-f86e5b076739" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/readyz\": dial tcp 10.134.0.25:8081: connect: connection refused" Oct 30 23:57:53.122567 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:53.122542 7570 generic.go:356] "Generic (PLEG): container finished" podID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" containerID="4c9d727f8c05504972593ae1b6240121490aba70db54418fa228de5c0ea97248" exitCode=0 Oct 30 23:57:53.122940 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:53.122917 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": dial tcp 10.134.0.157:8443: connect: connection refused" Oct 30 23:57:53.125167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:53.125144 7570 generic.go:356] "Generic (PLEG): container finished" podID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerID="437b364aa15062825ed4d26c6d17ec8b15ba4dd0916980db86c3f0f71ca026e9" exitCode=0 Oct 30 23:57:53.126877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:53.126853 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused" Oct 30 23:57:53.127126 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:53.127111 7570 generic.go:356] "Generic (PLEG): container finished" podID="b12dc177-11a1-40b0-9fdf-75993ab176e6" containerID="a2397d1a972d4c1e20267b88f3965c58ed65da73260e8e63c61ae66c513cbdab" exitCode=0 Oct 30 23:57:53.129110 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:53.129091 7570 generic.go:356] "Generic (PLEG): container finished" podID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" containerID="0170737f67453b691218a3069bf111fefb4d960cddcbbb0718f56f6f8b6dd3b2" exitCode=0 Oct 30 23:57:53.133539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:53.133518 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": dial tcp 10.134.0.131:8443: connect: connection refused" Oct 30 23:57:53.285212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:57:53.285181 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" podUID="d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.18:8081/readyz\": dial tcp 10.134.0.18:8081: connect: connection refused" Oct 30 23:58:13.716845 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:04.347444 7570 logging.go:55] [core] [Channel #3000 SubChannel #3012]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.171:50052: connect: connection refused" Oct 30 23:58:13.716845 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:57:53.370747 7570 logging.go:55] [core] [Channel #3006 SubChannel #3015]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.166:50052", ServerName: "10.134.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.166:50052: connect: connection refused" Oct 30 23:58:13.716845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.556987 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 23:58:13.716845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.666006 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": dial tcp 10.134.0.142:8443: connect: connection refused" Oct 30 23:58:13.716845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.666046 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-p5xwl" podUID="ee540c2e-f028-44ee-be53-d178b286fb02" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.24:9403/livez\": dial tcp 10.134.0.24:9403: connect: connection refused" Oct 30 23:58:13.726211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.666373 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": dial tcp 10.134.0.131:8443: connect: connection refused" Oct 30 23:58:13.726211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.666540 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": dial tcp 10.134.0.143:8443: connect: connection refused" Oct 30 23:58:13.726211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.666694 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": dial tcp 10.134.0.13:9091: connect: connection refused" start-of-body= Oct 30 23:58:13.726211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.666733 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": dial tcp 10.134.0.13:9091: connect: connection refused" Oct 30 23:58:13.726211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.667017 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": dial tcp 10.134.0.13:9091: connect: connection refused" start-of-body= Oct 30 23:58:13.726211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.667040 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": dial tcp 10.134.0.13:9091: connect: connection refused" Oct 30 23:58:13.726592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.667103 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": dial tcp 10.134.0.151:8443: connect: connection refused" Oct 30 23:58:13.726592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.691092 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 23:58:13.726592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.691654 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" podUID="b12dc177-11a1-40b0-9fdf-75993ab176e6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.166:50052\" within 1s: context deadline exceeded" Oct 30 23:58:13.726592 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.692165 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" podUID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.155:50052\" within 1s: context deadline exceeded" Oct 30 23:58:13.726592 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:04.693350 7570 logging.go:55] [core] [Channel #3045 SubChannel #3046]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.155:50052", ServerName: "10.134.0.155:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.155:50052: connect: connection refused" Oct 30 23:58:13.726846 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:04.693553 7570 logging.go:55] [core] [Channel #3043 SubChannel #3044]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.166:50052", ServerName: "10.134.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.166:50052: connect: connection refused" Oct 30 23:58:13.726846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.697018 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": dial tcp 10.134.0.135:8443: connect: connection refused" Oct 30 23:58:13.726846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.697132 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": dial tcp 10.134.0.157:8443: connect: connection refused" Oct 30 23:58:13.726846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.697211 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": dial tcp 10.134.0.135:8443: connect: connection refused" Oct 30 23:58:13.726846 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.699323 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": dial tcp 10.134.0.157:8443: connect: connection refused" Oct 30 23:58:13.726988 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.700026 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 23:58:13.726988 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.700062 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" podUID="f68436c1-d900-4063-b853-cb9be78ceae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 23:58:13.726988 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.700774 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused" Oct 30 23:58:13.726988 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.701141 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused" Oct 30 23:58:13.726988 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.702364 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded" Oct 30 23:58:13.727124 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:04.703901 7570 logging.go:55] [core] [Channel #3051 SubChannel #3052]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.171:50052: connect: connection refused" Oct 30 23:58:13.727124 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:04.556190 7570 logging.go:55] [core] [Channel #3001 SubChannel #3021]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.156:50052", ServerName: "10.134.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.156:50052: connect: connection refused" Oct 30 23:58:13.727124 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:04.556316 7570 logging.go:55] [core] [Channel #3008 SubChannel #3014]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.155:50052", ServerName: "10.134.0.155:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.155:50052: connect: connection refused" Oct 30 23:58:13.727124 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.714068 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-4blbf" podUID="1793bf9b-92db-4dae-bfd6-f86e5b076739" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/readyz\": dial tcp 10.134.0.25:8081: connect: connection refused" Oct 30 23:58:13.727124 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.722850 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": dial tcp 10.134.0.47:8443: connect: connection refused" Oct 30 23:58:13.727944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.723128 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": dial tcp 10.134.0.47:8443: connect: connection refused" Oct 30 23:58:13.727944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.733978 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-p5xwl" podUID="ee540c2e-f028-44ee-be53-d178b286fb02" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.24:9403/livez\": dial tcp 10.134.0.24:9403: connect: connection refused" Oct 30 23:58:13.727944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.734035 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" podUID="d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.18:8081/readyz\": dial tcp 10.134.0.18:8081: connect: connection refused" Oct 30 23:58:13.727944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.734719 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": dial tcp 10.134.0.142:8443: connect: connection refused" Oct 30 23:58:13.727944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.735207 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" podUID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.156:50052\" within 1s: context deadline exceeded" Oct 30 23:58:13.728116 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:04.736743 7570 logging.go:55] [core] [Channel #3072 SubChannel #3073]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.156:50052", ServerName: "10.134.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.156:50052: connect: connection refused" Oct 30 23:58:13.728116 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.736810 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": dial tcp 10.134.0.131:8443: connect: connection refused" Oct 30 23:58:13.728116 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.755209 7570 patch_prober.go:29] interesting pod/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 23:58:13.728116 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.755247 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 23:58:13.728116 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.755317 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:9091/-/healthy\": dial tcp 10.134.0.13:9091: connect: connection refused" start-of-body= Oct 30 23:58:13.728262 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.755358 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/healthy\": dial tcp 10.134.0.13:9091: connect: connection refused" Oct 30 23:58:13.728262 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:04.763026 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-2b4009: configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:58:13.728262 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:04.763092 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume podName:b12dc177-11a1-40b0-9fdf-75993ab176e6 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:20.763066916 +0000 UTC m=+2312.282067998 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume") pod "payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" (UID: "b12dc177-11a1-40b0-9fdf-75993ab176e6") : configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:58:13.728262 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:04.763137 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:58:13.728383 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:04.763171 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume podName:aa08b804-5246-4fb8-a68c-cf7af9ba4a54 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:20.763161032 +0000 UTC m=+2312.282162113 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume") pod "payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" (UID: "aa08b804-5246-4fb8-a68c-cf7af9ba4a54") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:58:13.728383 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:04.763222 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-df3b5f: configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:58:13.728383 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:04.763263 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume podName:e9c9e5ed-908c-463d-8f4f-395a80ababdd nodeName:}" failed. No retries permitted until 2026-10-30 23:58:20.763245687 +0000 UTC m=+2312.282246755 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" (UID: "e9c9e5ed-908c-463d-8f4f-395a80ababdd") : configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:58:13.728383 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:04.763518 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:58:13.728541 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:04.763592 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume podName:b4e6db40-bf7c-4440-bed0-8ec6e70cab0f nodeName:}" failed. No retries permitted until 2026-10-30 23:58:20.763576862 +0000 UTC m=+2312.282577944 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" (UID: "b4e6db40-bf7c-4440-bed0-8ec6e70cab0f") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:58:13.728541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.771324 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": dial tcp 10.134.0.20:6080: connect: connection refused" Oct 30 23:58:13.728541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.771630 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": dial tcp 10.134.0.20:6080: connect: connection refused" Oct 30 23:58:13.728541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.772663 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:58:13.728541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.800694 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" podUID="522aa71b-47e4-44b1-825c-bf81305d21b4" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:58:13.728684 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:04.823141 7570 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@13-35016-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1017ca7a_a16e_44bd_af55_d8babaa94e01.slice/crio-conmon-0bde901d48fa5140c259012bee5dd70f0ef26c4ba298b70a93e1e9c3ff2d7acd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68436c1_d900_4063_b853_cb9be78ceae6.slice/crio-conmon-c30617db6774365f2605f645b02abd554fc4a67999922b27bc2763f1ad73c11a.scope\": RecentStats: unable to find data in memory cache]" Oct 30 23:58:13.728684 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:04.952766 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" podUID="5244915c-9573-489c-b1f8-e22948899df3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.162:50052\" within 1s: context deadline exceeded" Oct 30 23:58:13.728684 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:04.993624 7570 logging.go:55] [core] [Channel #3038 SubChannel #3078]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.162:50052", ServerName: "10.134.0.162:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:52120->10.134.0.162:50052: use of closed network connection" Oct 30 23:58:13.728684 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.072292 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": dial tcp 10.134.0.151:8443: connect: connection refused" Oct 30 23:58:13.728796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.131821 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" podUID="201fbcb8-8fed-44e3-b60d-4b89d8c114ca" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.138:50052\" within 1s: context deadline exceeded" Oct 30 23:58:13.728796 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:05.172640 7570 logging.go:55] [core] [Channel #3039 SubChannel #3067]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.138:50052", ServerName: "10.134.0.138:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:56596->10.134.0.138:50052: use of closed network connection" Oct 30 23:58:13.728796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.256139 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": dial tcp 10.134.0.143:8443: connect: connection refused" Oct 30 23:58:13.728796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.335680 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 23:58:13.728796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.338288 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.728933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.339798 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 23:58:13.728933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.339853 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.728933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.341365 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.728933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.341549 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.728933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.342932 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 23:58:13.729073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.343474 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:58:13.729073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.344792 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.346124 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.348493 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.348513 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.351124 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 23:58:13.729209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.358181 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.368447 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.370595 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729209 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.374750 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.374767 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.374968 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.377593 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" podUID="37d7f8cf-f311-4b98-a6b7-25a043ac78ae" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:58:13.729352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.378550 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" podUID="a4acfbb3-3ee2-4890-984a-2fc8a796bece" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:58:13.729352 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.379592 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" podUID="1e8a6ea5-1c91-462f-94fd-7136a48971e9" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:58:13.729497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.379600 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" podUID="fb14f0ca-115b-4fe6-afcd-6338df872573" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:58:13.729497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.379600 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" podUID="c070c544-43c5-4796-b385-6c4094a63624" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:58:13.729497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.380542 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" podUID="e0f6ac75-7dd2-44be-8b54-92ff018b8a3d" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:58:13.729497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.381569 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.381572 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" podUID="6843e367-692b-41f0-a9c6-7704d732f009" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:58:13.729497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.382544 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" podUID="08022df4-0b83-43af-9b14-b85ab9f83be3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:58:13.729664 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.428560 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" podUID="ca8613ba-3272-43d3-9661-33618f859cff" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 23:58:13.729664 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.666527 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729664 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.692137 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" podUID="b12dc177-11a1-40b0-9fdf-75993ab176e6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.166:50052\" within 1s: context deadline exceeded" Oct 30 23:58:13.729664 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.692343 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" podUID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.155:50052\" within 1s: context deadline exceeded" Oct 30 23:58:13.729664 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.693310 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 23:58:13.729805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.694392 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.702249 7570 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-g86k2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:58:13.729805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.702295 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" podUID="9c20fb11-2dcf-445e-96cd-4d97045a2ab9" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.15:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.702572 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded" Oct 30 23:58:13.729805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.714846 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729937 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.719508 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729937 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.720596 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Oct 30 23:58:13.729937 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.720615 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.729937 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.731140 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" podUID="d387eda1-c364-4443-ba0d-619368f57f28" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.132:50052\" within 1s: context deadline exceeded" Oct 30 23:58:13.729937 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.736038 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" podUID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.156:50052\" within 1s: context deadline exceeded" Oct 30 23:58:13.730070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.741554 7570 patch_prober.go:29] interesting pod/metrics-server-5946dc94c7-djbgv container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:10250/livez\": context deadline exceeded" start-of-body= Oct 30 23:58:13.730070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.741592 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" podUID="ed360b42-1a02-4474-9bf1-2117d1b438f1" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.14:10250/livez\": context deadline exceeded" Oct 30 23:58:13.730070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.752646 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 23:58:13.730070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.755015 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:58:13.730070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.765551 7570 patch_prober.go:29] interesting pod/jobset-controller-manager-57b79bd74c-mqnfk container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:58:13.730070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.765563 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.730224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:05.765587 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" podUID="f7c5040c-6dbc-424d-a258-94092e75a027" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.730224 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:05.771584 7570 logging.go:55] [core] [Channel #3069 SubChannel #3070]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.132:50052", ServerName: "10.134.0.132:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:50512->10.134.0.132:50052: use of closed network connection" Oct 30 23:58:13.730224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:06.351612 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.730224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:06.351913 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 30 23:58:13.730224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:06.351960 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 23:58:13.730361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:06.352182 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 23:58:13.730361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:06.352375 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 23:58:13.730361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:06.353579 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.730361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:06.354038 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.730361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:06.357582 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.730517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:06.361475 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:58:13.730517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:07.554584 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": dial tcp 10.134.0.13:9091: connect: connection refused" start-of-body= Oct 30 23:58:13.730517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:07.554644 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": dial tcp 10.134.0.13:9091: connect: connection refused" Oct 30 23:58:13.730517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:07.560135 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": dial tcp 10.134.0.20:6080: connect: connection refused" Oct 30 23:58:13.730517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:08.027825 7570 patch_prober.go:29] interesting pod/jobset-controller-manager-57b79bd74c-mqnfk container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.134.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:58:13.730517 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:08.027881 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" podUID="f7c5040c-6dbc-424d-a258-94092e75a027" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:13.730679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:08.122158 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": dial tcp 10.134.0.135:8443: connect: connection refused" Oct 30 23:58:13.730679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:08.123897 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": dial tcp 10.134.0.157:8443: connect: connection refused" Oct 30 23:58:13.730679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:08.127753 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused" Oct 30 23:58:13.730679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:08.133184 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": dial tcp 10.134.0.131:8443: connect: connection refused" Oct 30 23:58:13.730679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:08.134429 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": dial tcp 10.134.0.47:8443: connect: connection refused" Oct 30 23:58:13.730812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:08.136712 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": dial tcp 10.134.0.142:8443: connect: connection refused" Oct 30 23:58:13.730812 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:08.867454 7570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="18.273s" Oct 30 23:58:13.730812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.040879 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6"] Oct 30 23:58:13.730812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.040942 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" event={"ID":"baaa353f-43a5-43e5-a73c-92d1840ebdae","Type":"ContainerStarted","Data":"06d7d77c68a6d09ea8dbd35e8e756af864eb288bb19d68836f86705ae868e8f1"} Oct 30 23:58:13.730812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.040964 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:58:13.730812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.165960 7570 patch_prober.go:29] interesting pod/thanos-querier-5f65cc44c7-ljdt2 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": dial tcp 10.134.0.13:9091: connect: connection refused" start-of-body= Oct 30 23:58:13.730812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.165999 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" podUID="d189925d-3211-43c6-8bf0-e9aca119e00e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": dial tcp 10.134.0.13:9091: connect: connection refused" Oct 30 23:58:13.730986 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.188213 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": dial tcp 10.134.0.20:6080: connect: connection refused" Oct 30 23:58:13.730986 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.221610 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" containerID="cri-o://7b62593c2ba5d2d1053164bdc55539dce34edcf66c3bf756b2febfddb756f124" gracePeriod=30 Oct 30 23:58:13.730986 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.248563 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" event={"ID":"f7c5040c-6dbc-424d-a258-94092e75a027","Type":"ContainerStarted","Data":"604990056fd8273b4a5a0ffd329cc207f31f7d39b0aba87c11aa050da026cf39"} Oct 30 23:58:13.730986 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.248717 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl"] Oct 30 23:58:13.730986 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.248752 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-p5xwl" event={"ID":"ee540c2e-f028-44ee-be53-d178b286fb02","Type":"ContainerStarted","Data":"4c11fbf267366e2100976d1c67fa85b58fe166a42658ff988b54a58c0cf154db"} Oct 30 23:58:13.730986 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.248780 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" event={"ID":"d189925d-3211-43c6-8bf0-e9aca119e00e","Type":"ContainerStarted","Data":"8765daa8fc617867e671b3f360879847d330db168358f849cf1abd81f965b81e"} Oct 30 23:58:13.730986 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.248833 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:58:13.731166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.248845 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-68r7m" event={"ID":"347b9a5b-729f-419a-bce5-749ac4aca5b5","Type":"ContainerStarted","Data":"52d666e6ca6dc516261efc37e2c3ba1dd4036026d6c5e9ff056bd7d80277816b"} Oct 30 23:58:13.731166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.248857 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf"] Oct 30 23:58:13.731166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.249135 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" containerID="cri-o://ec286697b6cc7b25e983764c1e1a5728facf90554c840204c0c4c9e59148fe30" gracePeriod=30 Oct 30 23:58:13.731166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.249266 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" containerID="cri-o://29f5ecced49b3f5128454a5896504d057362df6f345169fd68da0bbb5fa23782" gracePeriod=30 Oct 30 23:58:13.731166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250400 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:58:13.731166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250442 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:58:13.731166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250475 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" event={"ID":"aa08b804-5246-4fb8-a68c-cf7af9ba4a54","Type":"ContainerDied","Data":"4c9d727f8c05504972593ae1b6240121490aba70db54418fa228de5c0ea97248"} Oct 30 23:58:13.731166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250508 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24"] Oct 30 23:58:13.731404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250550 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798"] Oct 30 23:58:13.731404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250581 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" event={"ID":"e9c9e5ed-908c-463d-8f4f-395a80ababdd","Type":"ContainerDied","Data":"437b364aa15062825ed4d26c6d17ec8b15ba4dd0916980db86c3f0f71ca026e9"} Oct 30 23:58:13.731404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250608 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9"] Oct 30 23:58:13.731404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250647 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" event={"ID":"b12dc177-11a1-40b0-9fdf-75993ab176e6","Type":"ContainerDied","Data":"a2397d1a972d4c1e20267b88f3965c58ed65da73260e8e63c61ae66c513cbdab"} Oct 30 23:58:13.731404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250677 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" event={"ID":"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f","Type":"ContainerDied","Data":"0170737f67453b691218a3069bf111fefb4d960cddcbbb0718f56f6f8b6dd3b2"} Oct 30 23:58:13.731404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250769 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:58:13.731404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250806 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8"] Oct 30 23:58:13.731404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250879 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:58:13.731404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250935 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:58:13.752360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250952 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll"] Oct 30 23:58:13.752360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.250977 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk"] Oct 30 23:58:13.752360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.251032 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" containerID="cri-o://afc2d23e0695cedbf256b4a7750b95ea9fcba96be1a1b86cbbbaa52d2c4a5cf5" gracePeriod=30 Oct 30 23:58:13.752360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.251073 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:58:13.752360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.251144 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:58:13.752360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.251171 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk"] Oct 30 23:58:13.752360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.251204 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws"] Oct 30 23:58:13.752360 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.251334 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" containerID="cri-o://2da79595c41ddcf813b52babb72238b7840eb347770a0979f860ea3c9781803d" gracePeriod=30 Oct 30 23:58:13.752633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.251411 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" podUID="d8235a15-af1b-4332-ac83-1aef5bfb1286" containerName="istio-proxy" containerID="cri-o://384a06f41c709dbd8e0b9f506ff9f630aa2e0aa1e4523c8ed6cabb86121c1c89" gracePeriod=30 Oct 30 23:58:13.752633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.251725 7570 patch_prober.go:29] interesting pod/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 23:58:13.752633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.251758 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 23:58:13.752633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.251840 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" containerID="cri-o://a484bb709e0b21eb10b3c20ec1ffbc432be3950bc50ca72732a27baf0121f5d4" gracePeriod=30 Oct 30 23:58:13.752633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252134 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" containerID="cri-o://6e10c215485e3d6ccb26655f26754e13b3bcda5a3f83836c46072df0e29cc25a" gracePeriod=30 Oct 30 23:58:13.752633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252179 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:58:13.752794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252228 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:58:13.752794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252259 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:58:13.752794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252274 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc"] Oct 30 23:58:13.752794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252351 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:58:13.752794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252560 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 30 23:58:13.752794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252576 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 30 23:58:13.752794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252586 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 30 23:58:13.752794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252598 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 30 23:58:13.752794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252618 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 30 23:58:13.752794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252633 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" Oct 30 23:58:13.752794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.252648 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:58:13.753087 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.253249 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-57b79bd74c-mqnfk" Oct 30 23:58:13.753087 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.304438 7570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-east-2.compute.internal" event="NodeReady" Oct 30 23:58:13.753087 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.400667 7570 generic.go:356] "Generic (PLEG): container finished" podID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerID="0bde901d48fa5140c259012bee5dd70f0ef26c4ba298b70a93e1e9c3ff2d7acd" exitCode=0 Oct 30 23:58:13.753087 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.400742 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" event={"ID":"1017ca7a-a16e-44bd-af55-d8babaa94e01","Type":"ContainerDied","Data":"0bde901d48fa5140c259012bee5dd70f0ef26c4ba298b70a93e1e9c3ff2d7acd"} Oct 30 23:58:13.753087 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:09.404561 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-89shx" podUID="d3eeb220-e1ba-46e2-94c8-2bb19e02dde9" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.468648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.660270 7570 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-g86k2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9443/health\": context deadline exceeded" start-of-body= Oct 30 23:58:28.468648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.936023 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" podUID="5244915c-9573-489c-b1f8-e22948899df3" containerName="payload-processing" containerID="cri-o://b3c07a3b101c4549f4210027e795ad239cf58b95d2ed5169b8e57030fb403144" gracePeriod=13 Oct 30 23:58:28.468648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.938832 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" podUID="a74a48f3-2b68-424b-9489-33e22dab2f2f" containerName="payload-pre-processing" containerID="cri-o://851e4c679073d2c695b1c06fe3c2af3ff196bc23efffd3ae3aa7db70b8c17ad8" gracePeriod=13 Oct 30 23:58:28.468648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.938971 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" podUID="12efb180-e881-40a3-9001-8a39a2ddd352" containerName="payload-pre-processing" containerID="cri-o://3f64b5a78b92f2c34f681907d33dd8081813506ac9e5ee82e9f44f06a2e6b2fd" gracePeriod=13 Oct 30 23:58:28.468648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.957839 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-7vq6t" podUID="d54e0641-5f0d-41bd-8b1b-900463a24cfa" containerName="extractor" probeResult="failure" output="command timed out" Oct 30 23:58:28.468648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.969105 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": dial tcp 10.134.0.47:8443: connect: connection refused" Oct 30 23:58:28.506351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.969182 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": dial tcp 10.134.0.131:8443: connect: connection refused" Oct 30 23:58:28.506351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.969315 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g86k2" podUID="9c20fb11-2dcf-445e-96cd-4d97045a2ab9" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.15:9443/health\": context deadline exceeded" Oct 30 23:58:28.506351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.969340 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": dial tcp 10.134.0.20:6080: connect: connection refused" Oct 30 23:58:28.506351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.969656 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused" Oct 30 23:58:28.506351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.969919 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.506575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.971750 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": dial tcp 10.134.0.157:8443: connect: connection refused" Oct 30 23:58:28.506575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.971830 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.506575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.971869 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-4blbf" podUID="1793bf9b-92db-4dae-bfd6-f86e5b076739" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/readyz\": dial tcp 10.134.0.25:8081: connect: connection refused" Oct 30 23:58:28.506575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.971933 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": dial tcp 10.134.0.135:8443: connect: connection refused" Oct 30 23:58:28.506575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.971910 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 23:58:28.506762 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.971980 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.506762 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.972034 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 23:58:28.506762 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.972155 7570 patch_prober.go:29] interesting pod/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.506762 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.972206 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" podUID="d8235a15-af1b-4332-ac83-1aef5bfb1286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 23:58:28.506762 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:26.972335 7570 logging.go:55] [core] [Channel #3148 SubChannel #3150]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.171:50052: connect: connection refused" Oct 30 23:58:28.507033 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.972447 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.507033 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.977302 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 23:58:28.507033 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.974387 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-p5xwl" podUID="ee540c2e-f028-44ee-be53-d178b286fb02" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.24:9403/livez\": dial tcp 10.134.0.24:9403: connect: connection refused" Oct 30 23:58:28.507033 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.976073 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": dial tcp 10.134.0.143:8443: connect: connection refused" Oct 30 23:58:28.507033 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.976136 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": dial tcp 10.134.0.151:8443: connect: connection refused" Oct 30 23:58:28.507224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.976668 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": dial tcp 10.134.0.142:8443: connect: connection refused" Oct 30 23:58:28.507224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.973133 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" podUID="d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.18:8081/readyz\": dial tcp 10.134.0.18:8081: connect: connection refused" Oct 30 23:58:28.507224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.973365 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.507224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.977438 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 23:58:28.507224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.988928 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" podUID="fb14f0ca-115b-4fe6-afcd-6338df872573" containerName="payload-processing" containerID="cri-o://734e0129e0d5009c2d894c51431c9a5313ff3f54071b2dc5309f546eade589d3" gracePeriod=13 Oct 30 23:58:28.507437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.992914 7570 patch_prober.go:29] interesting pod/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.507437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:26.992949 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 23:58:28.507437 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:26.997752 7570 logging.go:55] [core] [Channel #3149 SubChannel #3159]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.156:50052", ServerName: "10.134.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.156:50052: connect: connection refused" Oct 30 23:58:28.507437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.003193 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.507437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.003230 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 23:58:28.507674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.003718 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.507674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.003750 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 23:58:28.507674 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:27.003996 7570 logging.go:55] [core] [Channel #3145 SubChannel #3162]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.166:50052", ServerName: "10.134.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.166:50052: connect: connection refused" Oct 30 23:58:28.507674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.005315 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.507674 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.005366 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 23:58:28.507939 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:27.016053 7570 logging.go:55] [core] [Channel #3139 SubChannel #3168]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.155:50052", ServerName: "10.134.0.155:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.155:50052: connect: connection refused" Oct 30 23:58:28.507939 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.017008 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.507939 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.023594 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" podUID="f68436c1-d900-4063-b853-cb9be78ceae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 23:58:28.507939 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.028994 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="da3a65c4-b342-42d7-aad0-29e63b90125a" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 23:58:28.507939 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.029077 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="da3a65c4-b342-42d7-aad0-29e63b90125a" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 23:58:28.507939 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:27.030531 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:58:28.508342 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:27.030597 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume podName:aa08b804-5246-4fb8-a68c-cf7af9ba4a54 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:59.030571032 +0000 UTC m=+2350.549572118 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume") pod "payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" (UID: "aa08b804-5246-4fb8-a68c-cf7af9ba4a54") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:58:28.508342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.038626 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 23:58:28.508342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.059754 7570 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mxggs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:58:28.508342 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.059792 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" podUID="363febe9-9102-48c9-8faa-09bf49151bf0" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.116:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.508342 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:27.061220 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-2b4009: configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:58:28.508695 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:27.061285 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume podName:b12dc177-11a1-40b0-9fdf-75993ab176e6 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:59.0612643 +0000 UTC m=+2350.580265382 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume") pod "payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" (UID: "b12dc177-11a1-40b0-9fdf-75993ab176e6") : configmap "payload-processing-plugins-e2e-inf-a-2b4009" not found Oct 30 23:58:28.508695 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:27.061334 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-df3b5f: configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:58:28.508695 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:27.061369 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume podName:e9c9e5ed-908c-463d-8f4f-395a80ababdd nodeName:}" failed. No retries permitted until 2026-10-30 23:58:59.061357783 +0000 UTC m=+2350.580358865 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" (UID: "e9c9e5ed-908c-463d-8f4f-395a80ababdd") : configmap "payload-processing-plugins-e2e-inf-b-df3b5f" not found Oct 30 23:58:28.508695 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:27.062125 7570 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-82d449: configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:58:28.508936 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:27.062188 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume podName:b4e6db40-bf7c-4440-bed0-8ec6e70cab0f nodeName:}" failed. No retries permitted until 2026-10-30 23:58:59.062173076 +0000 UTC m=+2350.581174163 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" (UID: "b4e6db40-bf7c-4440-bed0-8ec6e70cab0f") : configmap "payload-processing-plugins-e2e-inf-a-82d449" not found Oct 30 23:58:28.509062 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:27.260932 7570 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ec9765_6ba5_4e0a_ba9f_3ae35dbb7984.slice/crio-conmon-7b62593c2ba5d2d1053164bdc55539dce34edcf66c3bf756b2febfddb756f124.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8235a15_af1b_4332_ac83_1aef5bfb1286.slice/crio-conmon-384a06f41c709dbd8e0b9f506ff9f630aa2e0aa1e4523c8ed6cabb86121c1c89.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf179a0a0_c464_4a8b_b6cc_04a21610a832.slice/crio-29f5ecced49b3f5128454a5896504d057362df6f345169fd68da0bbb5fa23782.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf179a0a0_c464_4a8b_b6cc_04a21610a832.slice/crio-conmon-29f5ecced49b3f5128454a5896504d057362df6f345169fd68da0bbb5fa23782.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f473a6_9517_4367_b935_294dbd28ceec.slice/crio-ec286697b6cc7b25e983764c1e1a5728facf90554c840204c0c4c9e59148fe30.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e446a0_c6da_40ab_aa1f_f3756e39619a.slice/crio-2da79595c41ddcf813b52babb72238b7840eb347770a0979f860ea3c9781803d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fc6c4a_5ced_4748_9d6f_c7cede8c7dc7.slice/crio-a484bb709e0b21eb10b3c20ec1ffbc432be3950bc50ca72732a27baf0121f5d4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f473a6_9517_4367_b935_294dbd28ceec.slice/crio-conmon-ec286697b6cc7b25e983764c1e1a5728facf90554c840204c0c4c9e59148fe30.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94094c6_876a_4082_ad7d_f35bb1909f87.slice/crio-conmon-afc2d23e0695cedbf256b4a7750b95ea9fcba96be1a1b86cbbbaa52d2c4a5cf5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e446a0_c6da_40ab_aa1f_f3756e39619a.slice/crio-conmon-2da79595c41ddcf813b52babb72238b7840eb347770a0979f860ea3c9781803d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94094c6_876a_4082_ad7d_f35bb1909f87.slice/crio-afc2d23e0695cedbf256b4a7750b95ea9fcba96be1a1b86cbbbaa52d2c4a5cf5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ec9765_6ba5_4e0a_ba9f_3ae35dbb7984.slice/crio-7b62593c2ba5d2d1053164bdc55539dce34edcf66c3bf756b2febfddb756f124.scope\": RecentStats: unable to find data in memory cache]" Oct 30 23:58:28.509062 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:27.270746 7570 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ec9765_6ba5_4e0a_ba9f_3ae35dbb7984.slice/crio-conmon-7b62593c2ba5d2d1053164bdc55539dce34edcf66c3bf756b2febfddb756f124.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94094c6_876a_4082_ad7d_f35bb1909f87.slice/crio-conmon-afc2d23e0695cedbf256b4a7750b95ea9fcba96be1a1b86cbbbaa52d2c4a5cf5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f473a6_9517_4367_b935_294dbd28ceec.slice/crio-conmon-ec286697b6cc7b25e983764c1e1a5728facf90554c840204c0c4c9e59148fe30.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf179a0a0_c464_4a8b_b6cc_04a21610a832.slice/crio-conmon-29f5ecced49b3f5128454a5896504d057362df6f345169fd68da0bbb5fa23782.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fc6c4a_5ced_4748_9d6f_c7cede8c7dc7.slice/crio-conmon-a484bb709e0b21eb10b3c20ec1ffbc432be3950bc50ca72732a27baf0121f5d4.scope\": RecentStats: unable to find data in memory cache]" Oct 30 23:58:28.509062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.659481 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.509062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.659493 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.509062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.659728 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.509062 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.660629 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.768227 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.771362 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.772546 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" podUID="ca8613ba-3272-43d3-9661-33618f859cff" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.139:50052\" within 1s: context deadline exceeded" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:27.802671 7570 logging.go:55] [core] [Channel #3154 SubChannel #3155]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.139:50052", ServerName: "10.134.0.139:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:45392->10.134.0.139:50052: use of closed network connection" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.898876 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.899397 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.899500 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.901890 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.903804 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.905856 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.918018 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" podUID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.155:50052\" within 1s: context deadline exceeded" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.931088 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.938254 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-c6888" podUID="eeffad66-ea6b-486f-abd8-63e68120af4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.939425 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" podUID="b12dc177-11a1-40b0-9fdf-75993ab176e6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.166:50052\" within 1s: context deadline exceeded" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.943510 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.509316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.948839 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.949923 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.951066 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.951164 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" podUID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.156:50052\" within 1s: context deadline exceeded" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.957588 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.962826 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.964881 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.965947 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:27.968359 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.022409 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.022430 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx" podUID="dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.025763 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.025952 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2" podUID="5275c1e3-85ea-449f-b526-7c3cbb14ed4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.030741 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.038933 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" podUID="f87cf2ab-c80c-4680-9c10-497e2a11235b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.139632 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": dial tcp 10.134.0.135:8443: connect: connection refused" Oct 30 23:58:28.510156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.140515 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.140580 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded" Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.140657 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused" Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.140688 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": dial tcp 10.134.0.157:8443: connect: connection refused" Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.141492 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.141529 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" podUID="f68436c1-d900-4063-b853-cb9be78ceae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.144482 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": dial tcp 10.134.0.131:8443: connect: connection refused" Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.145179 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": dial tcp 10.134.0.47:8443: connect: connection refused" Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.148701 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.148734 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.157197 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": dial tcp 10.134.0.142:8443: connect: connection refused" Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.185031 7570 patch_prober.go:29] interesting pod/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.185069 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" podUID="d8235a15-af1b-4332-ac83-1aef5bfb1286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.186532 7570 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.186562 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.186695 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.186716 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 23:58:28.510969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.187947 7570 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.511876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.187983 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 23:58:28.511876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.188044 7570 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.511876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.188064 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 23:58:28.511876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.196091 7570 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.511876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.196126 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 23:58:28.511876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.279332 7570 patch_prober.go:29] interesting pod/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 23:58:28.511876 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.279378 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 23:58:28.762819 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:28.762747 7570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.722s" Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.812438 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.812530 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f65cc44c7-ljdt2" Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.812686 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.812716 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.812747 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.812775 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.812794 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj"] Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.812846 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.812872 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6pxqs" Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.813168 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerName="istio-proxy" containerID="cri-o://041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931" gracePeriod=30 Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.831999 7570 generic.go:356] "Generic (PLEG): container finished" podID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerID="6e10c215485e3d6ccb26655f26754e13b3bcda5a3f83836c46072df0e29cc25a" exitCode=0 Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.832072 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" event={"ID":"af49f007-5f21-43cc-b4f3-d78e31b45100","Type":"ContainerDied","Data":"6e10c215485e3d6ccb26655f26754e13b3bcda5a3f83836c46072df0e29cc25a"} Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.998667 7570 generic.go:356] "Generic (PLEG): container finished" podID="f68436c1-d900-4063-b853-cb9be78ceae6" containerID="c30617db6774365f2605f645b02abd554fc4a67999922b27bc2763f1ad73c11a" exitCode=0 Oct 30 23:58:29.062809 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:28.998731 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" event={"ID":"f68436c1-d900-4063-b853-cb9be78ceae6","Type":"ContainerDied","Data":"c30617db6774365f2605f645b02abd554fc4a67999922b27bc2763f1ad73c11a"} Oct 30 23:58:29.189632 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:29.189598 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": dial tcp 10.134.0.20:6080: connect: connection refused" Oct 30 23:58:30.056574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:30.054134 7570 generic.go:356] "Generic (PLEG): container finished" podID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerID="29f5ecced49b3f5128454a5896504d057362df6f345169fd68da0bbb5fa23782" exitCode=0 Oct 30 23:58:30.056574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:30.054314 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" event={"ID":"f179a0a0-c464-4a8b-b6cc-04a21610a832","Type":"ContainerDied","Data":"29f5ecced49b3f5128454a5896504d057362df6f345169fd68da0bbb5fa23782"} Oct 30 23:58:30.056574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:30.054385 7570 scope.go:122] "RemoveContainer" containerID="6240555c4a0f290fefac4ba05d496628c7c52e564af57f302251937420a0f092" Oct 30 23:58:30.056574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:30.056440 7570 generic.go:356] "Generic (PLEG): container finished" podID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerID="afc2d23e0695cedbf256b4a7750b95ea9fcba96be1a1b86cbbbaa52d2c4a5cf5" exitCode=0 Oct 30 23:58:30.057372 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:30.057266 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" event={"ID":"b94094c6-876a-4082-ad7d-f35bb1909f87","Type":"ContainerDied","Data":"afc2d23e0695cedbf256b4a7750b95ea9fcba96be1a1b86cbbbaa52d2c4a5cf5"} Oct 30 23:58:30.076663 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:30.069954 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:58:30.076663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:30.073308 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": dial tcp 10.134.0.151:8443: connect: connection refused" Oct 30 23:58:30.076663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:30.074114 7570 generic.go:356] "Generic (PLEG): container finished" podID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerID="7b62593c2ba5d2d1053164bdc55539dce34edcf66c3bf756b2febfddb756f124" exitCode=0 Oct 30 23:58:30.076663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:30.074183 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" event={"ID":"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984","Type":"ContainerDied","Data":"7b62593c2ba5d2d1053164bdc55539dce34edcf66c3bf756b2febfddb756f124"} Oct 30 23:58:30.076663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:30.076288 7570 generic.go:356] "Generic (PLEG): container finished" podID="a6f473a6-9517-4367-b935-294dbd28ceec" containerID="ec286697b6cc7b25e983764c1e1a5728facf90554c840204c0c4c9e59148fe30" exitCode=0 Oct 30 23:58:30.081678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:30.077806 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" event={"ID":"a6f473a6-9517-4367-b935-294dbd28ceec","Type":"ContainerDied","Data":"ec286697b6cc7b25e983764c1e1a5728facf90554c840204c0c4c9e59148fe30"} Oct 30 23:58:30.081678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:30.081610 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" podUID="baaa353f-43a5-43e5-a73c-92d1840ebdae" containerName="cert-manager-webhook" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 30 23:58:30.270230 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:30.260587 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": dial tcp 10.134.0.143:8443: connect: connection refused" Oct 30 23:58:31.005700 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.005668 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:58:31.094361 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.093905 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:58:31.102132 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.102054 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:58:31.105824 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.105799 7570 generic.go:356] "Generic (PLEG): container finished" podID="a74a48f3-2b68-424b-9489-33e22dab2f2f" containerID="851e4c679073d2c695b1c06fe3c2af3ff196bc23efffd3ae3aa7db70b8c17ad8" exitCode=0 Oct 30 23:58:31.106070 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.106048 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" event={"ID":"a74a48f3-2b68-424b-9489-33e22dab2f2f","Type":"ContainerDied","Data":"851e4c679073d2c695b1c06fe3c2af3ff196bc23efffd3ae3aa7db70b8c17ad8"} Oct 30 23:58:31.109271 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.109149 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c9e5ed-908c-463d-8f4f-395a80ababdd-kube-api-access-snm8f\" (UniqueName: \"kubernetes.io/projected/e9c9e5ed-908c-463d-8f4f-395a80ababdd-kube-api-access-snm8f\") pod \"e9c9e5ed-908c-463d-8f4f-395a80ababdd\" (UID: \"e9c9e5ed-908c-463d-8f4f-395a80ababdd\") " Oct 30 23:58:31.109601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.109580 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume\") pod \"e9c9e5ed-908c-463d-8f4f-395a80ababdd\" (UID: \"e9c9e5ed-908c-463d-8f4f-395a80ababdd\") " Oct 30 23:58:31.111644 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.111615 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume" pod "e9c9e5ed-908c-463d-8f4f-395a80ababdd" (UID: "e9c9e5ed-908c-463d-8f4f-395a80ababdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:31.111805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.111788 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:58:31.111964 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.111797 7570 generic.go:356] "Generic (PLEG): container finished" podID="fb14f0ca-115b-4fe6-afcd-6338df872573" containerID="734e0129e0d5009c2d894c51431c9a5313ff3f54071b2dc5309f546eade589d3" exitCode=0 Oct 30 23:58:31.112167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.111816 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" event={"ID":"fb14f0ca-115b-4fe6-afcd-6338df872573","Type":"ContainerDied","Data":"734e0129e0d5009c2d894c51431c9a5313ff3f54071b2dc5309f546eade589d3"} Oct 30 23:58:31.114730 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.114704 7570 generic.go:356] "Generic (PLEG): container finished" podID="d8235a15-af1b-4332-ac83-1aef5bfb1286" containerID="384a06f41c709dbd8e0b9f506ff9f630aa2e0aa1e4523c8ed6cabb86121c1c89" exitCode=0 Oct 30 23:58:31.114821 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.114794 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" event={"ID":"d8235a15-af1b-4332-ac83-1aef5bfb1286","Type":"ContainerDied","Data":"384a06f41c709dbd8e0b9f506ff9f630aa2e0aa1e4523c8ed6cabb86121c1c89"} Oct 30 23:58:31.115511 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.115453 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:58:31.117114 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.117095 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" Oct 30 23:58:31.118443 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.118423 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c" event={"ID":"e9c9e5ed-908c-463d-8f4f-395a80ababdd","Type":"ContainerDied","Data":"b148fbb0cc25ede32e927038e6425249f74bfc965590f261c078e6de63cf1bd2"} Oct 30 23:58:31.123136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.123114 7570 generic.go:356] "Generic (PLEG): container finished" podID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerID="2da79595c41ddcf813b52babb72238b7840eb347770a0979f860ea3c9781803d" exitCode=0 Oct 30 23:58:31.123365 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.123346 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" event={"ID":"c9e446a0-c6da-40ab-aa1f-f3756e39619a","Type":"ContainerDied","Data":"2da79595c41ddcf813b52babb72238b7840eb347770a0979f860ea3c9781803d"} Oct 30 23:58:31.130708 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.130688 7570 generic.go:356] "Generic (PLEG): container finished" podID="5244915c-9573-489c-b1f8-e22948899df3" containerID="b3c07a3b101c4549f4210027e795ad239cf58b95d2ed5169b8e57030fb403144" exitCode=0 Oct 30 23:58:31.130799 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.130758 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" event={"ID":"5244915c-9573-489c-b1f8-e22948899df3","Type":"ContainerDied","Data":"b3c07a3b101c4549f4210027e795ad239cf58b95d2ed5169b8e57030fb403144"} Oct 30 23:58:31.132503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.132480 7570 generic.go:356] "Generic (PLEG): container finished" podID="12efb180-e881-40a3-9001-8a39a2ddd352" containerID="3f64b5a78b92f2c34f681907d33dd8081813506ac9e5ee82e9f44f06a2e6b2fd" exitCode=0 Oct 30 23:58:31.132593 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.132510 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" event={"ID":"12efb180-e881-40a3-9001-8a39a2ddd352","Type":"ContainerDied","Data":"3f64b5a78b92f2c34f681907d33dd8081813506ac9e5ee82e9f44f06a2e6b2fd"} Oct 30 23:58:31.134061 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.134042 7570 generic.go:356] "Generic (PLEG): container finished" podID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerID="a484bb709e0b21eb10b3c20ec1ffbc432be3950bc50ca72732a27baf0121f5d4" exitCode=0 Oct 30 23:58:31.134138 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.134104 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" event={"ID":"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7","Type":"ContainerDied","Data":"a484bb709e0b21eb10b3c20ec1ffbc432be3950bc50ca72732a27baf0121f5d4"} Oct 30 23:58:31.135399 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.135373 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" Oct 30 23:58:31.135513 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.135383 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr" event={"ID":"1017ca7a-a16e-44bd-af55-d8babaa94e01","Type":"ContainerDied","Data":"3e05a8a2136a85be0af92165b28087911cd43e23940b0abc5b811e21c627e267"} Oct 30 23:58:31.138115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.138096 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:58:31.139281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.139262 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" Oct 30 23:58:31.139516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.139284 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6" event={"ID":"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984","Type":"ContainerDied","Data":"c218cb1a1f5e8bf30f515d0d4dfbde6481fb1e84dfc8100da0b7f1098701f14c"} Oct 30 23:58:31.141155 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.141136 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" Oct 30 23:58:31.141316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.141156 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght" event={"ID":"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f","Type":"ContainerDied","Data":"4b86cd137d561736b49011f33a2776b4b31a08a41a2dd50f166c30ae6407bbc8"} Oct 30 23:58:31.211338 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.211272 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-data\") pod \"1017ca7a-a16e-44bd-af55-d8babaa94e01\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " Oct 30 23:58:31.211522 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.211448 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-certs\") pod \"1017ca7a-a16e-44bd-af55-d8babaa94e01\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " Oct 30 23:58:31.211614 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.211552 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-kube-api-access-vpzk6\" (UniqueName: \"kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-kube-api-access-vpzk6\") pod \"1017ca7a-a16e-44bd-af55-d8babaa94e01\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " Oct 30 23:58:31.211671 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.211645 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-socket\") pod \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " Oct 30 23:58:31.211726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.211707 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-certs\") pod \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " Oct 30 23:58:31.211786 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.211771 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-kube-api-access-m875z\" (UniqueName: \"kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-kube-api-access-m875z\") pod \"b94094c6-876a-4082-ad7d-f35bb1909f87\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " Oct 30 23:58:31.211853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.211798 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-token\") pod \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " Oct 30 23:58:31.211853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.211838 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-kube-api-access-9q5t4\" (UniqueName: \"kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-kube-api-access-9q5t4\") pod \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " Oct 30 23:58:31.211930 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.211916 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-socket\") pod \"1017ca7a-a16e-44bd-af55-d8babaa94e01\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " Oct 30 23:58:31.212019 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212003 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume\") pod \"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f\" (UID: \"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f\") " Oct 30 23:58:31.212086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212070 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istiod-ca-cert\") pod \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " Oct 30 23:58:31.212150 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212131 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-envoy\") pod \"1017ca7a-a16e-44bd-af55-d8babaa94e01\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " Oct 30 23:58:31.212199 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212171 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-certs\") pod \"b94094c6-876a-4082-ad7d-f35bb1909f87\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " Oct 30 23:58:31.212246 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212231 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-credential-socket\") pod \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " Oct 30 23:58:31.212298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.211309 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log" Oct 30 23:58:31.212298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212259 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-credential-socket\") pod \"1017ca7a-a16e-44bd-af55-d8babaa94e01\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " Oct 30 23:58:31.212380 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212329 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-token\" (UniqueName: \"kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-token\") pod \"1017ca7a-a16e-44bd-af55-d8babaa94e01\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " Oct 30 23:58:31.212421 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212393 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-credential-socket\") pod \"b94094c6-876a-4082-ad7d-f35bb1909f87\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " Oct 30 23:58:31.212558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212541 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b94094c6-876a-4082-ad7d-f35bb1909f87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94094c6-876a-4082-ad7d-f35bb1909f87-istiod-ca-cert\") pod \"b94094c6-876a-4082-ad7d-f35bb1909f87\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " Oct 30 23:58:31.212616 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212562 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:58:31.212652 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212613 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-podinfo\") pod \"1017ca7a-a16e-44bd-af55-d8babaa94e01\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " Oct 30 23:58:31.212695 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212647 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-data\") pod \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " Oct 30 23:58:31.212695 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212677 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1017ca7a-a16e-44bd-af55-d8babaa94e01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1017ca7a-a16e-44bd-af55-d8babaa94e01-istiod-ca-cert\") pod \"1017ca7a-a16e-44bd-af55-d8babaa94e01\" (UID: \"1017ca7a-a16e-44bd-af55-d8babaa94e01\") " Oct 30 23:58:31.212766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212711 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-data\") pod \"b94094c6-876a-4082-ad7d-f35bb1909f87\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " Oct 30 23:58:31.212806 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212791 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-podinfo\") pod \"b94094c6-876a-4082-ad7d-f35bb1909f87\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " Oct 30 23:58:31.212869 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212849 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-socket\") pod \"b94094c6-876a-4082-ad7d-f35bb1909f87\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " Oct 30 23:58:31.212947 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212879 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-podinfo\") pod \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " Oct 30 23:58:31.212947 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212914 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-kube-api-access-vpr7l\" (UniqueName: \"kubernetes.io/projected/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-kube-api-access-vpr7l\") pod \"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f\" (UID: \"b4e6db40-bf7c-4440-bed0-8ec6e70cab0f\") " Oct 30 23:58:31.213035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.212945 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-token\" (UniqueName: \"kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-token\") pod \"b94094c6-876a-4082-ad7d-f35bb1909f87\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " Oct 30 23:58:31.213035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.213011 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-envoy\") pod \"b94094c6-876a-4082-ad7d-f35bb1909f87\" (UID: \"b94094c6-876a-4082-ad7d-f35bb1909f87\") " Oct 30 23:58:31.213100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.213035 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-envoy\") pod \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\" (UID: \"e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984\") " Oct 30 23:58:31.213307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.213280 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" event={"ID":"29c3d1e4-09b5-4712-ad6d-9a32434def2d","Type":"ContainerStarted","Data":"3139737f6000f047ae360c707bc9c3b452855c725d1ca3b7c30a64c3df5d344b"} Oct 30 23:58:31.214647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.214624 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c9e5ed-908c-463d-8f4f-395a80ababdd-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.226442 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.226413 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume" pod "b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" (UID: "b4e6db40-bf7c-4440-bed0-8ec6e70cab0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:31.303184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.302279 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-credential-socket" pod "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" (UID: "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.303184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.303105 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-data" pod "b94094c6-876a-4082-ad7d-f35bb1909f87" (UID: "b94094c6-876a-4082-ad7d-f35bb1909f87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.308191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.308142 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-data" pod "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" (UID: "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.309519 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.309492 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-credential-socket" pod "b94094c6-876a-4082-ad7d-f35bb1909f87" (UID: "b94094c6-876a-4082-ad7d-f35bb1909f87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.309638 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.309561 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-certs" pod "b94094c6-876a-4082-ad7d-f35bb1909f87" (UID: "b94094c6-876a-4082-ad7d-f35bb1909f87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.309717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.309699 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94094c6-876a-4082-ad7d-f35bb1909f87-istiod-ca-cert" pod "b94094c6-876a-4082-ad7d-f35bb1909f87" (UID: "b94094c6-876a-4082-ad7d-f35bb1909f87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:31.313007 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.312983 7570 scope.go:122] "RemoveContainer" containerID="8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50" Oct 30 23:58:31.335167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.316598 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-socket" pod "b94094c6-876a-4082-ad7d-f35bb1909f87" (UID: "b94094c6-876a-4082-ad7d-f35bb1909f87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.335167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.317202 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-data\") pod \"a6f473a6-9517-4367-b935-294dbd28ceec\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " Oct 30 23:58:31.335167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.317307 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6f473a6-9517-4367-b935-294dbd28ceec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f473a6-9517-4367-b935-294dbd28ceec-istio-podinfo\") pod \"a6f473a6-9517-4367-b935-294dbd28ceec\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " Oct 30 23:58:31.335167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.317438 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-envoy\") pod \"a6f473a6-9517-4367-b935-294dbd28ceec\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " Oct 30 23:58:31.335167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.317538 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-certs\") pod \"a6f473a6-9517-4367-b935-294dbd28ceec\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " Oct 30 23:58:31.335167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.317588 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f473a6-9517-4367-b935-294dbd28ceec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f473a6-9517-4367-b935-294dbd28ceec-istiod-ca-cert\") pod \"a6f473a6-9517-4367-b935-294dbd28ceec\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " Oct 30 23:58:31.335167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.317693 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-istio-token\" (UniqueName: \"kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-istio-token\") pod \"a6f473a6-9517-4367-b935-294dbd28ceec\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " Oct 30 23:58:31.335167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.317892 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-socket\") pod \"a6f473a6-9517-4367-b935-294dbd28ceec\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " Oct 30 23:58:31.335167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.318149 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-credential-socket\") pod \"a6f473a6-9517-4367-b935-294dbd28ceec\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " Oct 30 23:58:31.335167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.318278 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-kube-api-access-gplkm\" (UniqueName: \"kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-kube-api-access-gplkm\") pod \"a6f473a6-9517-4367-b935-294dbd28ceec\" (UID: \"a6f473a6-9517-4367-b935-294dbd28ceec\") " Oct 30 23:58:31.338113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.337327 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.338113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.337369 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.338113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.337389 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.338113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.337406 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.338113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.337421 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94094c6-876a-4082-ad7d-f35bb1909f87-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.338113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.337433 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.338113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.337448 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.338113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.337482 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.420739 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.420370 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-socket" pod "1017ca7a-a16e-44bd-af55-d8babaa94e01" (UID: "1017ca7a-a16e-44bd-af55-d8babaa94e01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.423489 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.421348 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-credential-socket" pod "1017ca7a-a16e-44bd-af55-d8babaa94e01" (UID: "1017ca7a-a16e-44bd-af55-d8babaa94e01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.464129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.454323 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-socket" pod "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" (UID: "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.464129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.460247 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.464129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.460280 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.464129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.460297 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.464129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.461090 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c9e5ed-908c-463d-8f4f-395a80ababdd-kube-api-access-snm8f" pod "e9c9e5ed-908c-463d-8f4f-395a80ababdd" (UID: "e9c9e5ed-908c-463d-8f4f-395a80ababdd"). InnerVolumeSpecName "kube-api-access-snm8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:31.496546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.496510 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-podinfo" pod "b94094c6-876a-4082-ad7d-f35bb1909f87" (UID: "b94094c6-876a-4082-ad7d-f35bb1909f87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:58:31.496628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.496608 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-certs" pod "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" (UID: "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.497212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.497174 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-podinfo" pod "1017ca7a-a16e-44bd-af55-d8babaa94e01" (UID: "1017ca7a-a16e-44bd-af55-d8babaa94e01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:58:31.497312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.497226 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-envoy" pod "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" (UID: "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.497532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.497504 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-kube-api-access-vpr7l" pod "b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" (UID: "b4e6db40-bf7c-4440-bed0-8ec6e70cab0f"). InnerVolumeSpecName "kube-api-access-vpr7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:31.498633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.498608 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-kube-api-access-vpzk6" pod "1017ca7a-a16e-44bd-af55-d8babaa94e01" (UID: "1017ca7a-a16e-44bd-af55-d8babaa94e01"). InnerVolumeSpecName "kube-api-access-vpzk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:31.498720 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.498651 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-token" pod "1017ca7a-a16e-44bd-af55-d8babaa94e01" (UID: "1017ca7a-a16e-44bd-af55-d8babaa94e01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:31.498844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.498815 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-podinfo" pod "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" (UID: "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:58:31.498901 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.498839 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-envoy" pod "b94094c6-876a-4082-ad7d-f35bb1909f87" (UID: "b94094c6-876a-4082-ad7d-f35bb1909f87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.498901 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.498867 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-kube-api-access-m875z" pod "b94094c6-876a-4082-ad7d-f35bb1909f87" (UID: "b94094c6-876a-4082-ad7d-f35bb1909f87"). InnerVolumeSpecName "kube-api-access-m875z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:31.498981 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.498917 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-token" pod "b94094c6-876a-4082-ad7d-f35bb1909f87" (UID: "b94094c6-876a-4082-ad7d-f35bb1909f87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:31.499013 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.498999 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-token" pod "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" (UID: "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:31.499073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.499057 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-kube-api-access-9q5t4" pod "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" (UID: "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984"). InnerVolumeSpecName "kube-api-access-9q5t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:31.530286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.530227 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-data" pod "1017ca7a-a16e-44bd-af55-d8babaa94e01" (UID: "1017ca7a-a16e-44bd-af55-d8babaa94e01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.530432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.530303 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-certs" pod "1017ca7a-a16e-44bd-af55-d8babaa94e01" (UID: "1017ca7a-a16e-44bd-af55-d8babaa94e01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.530432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.530350 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-envoy" pod "1017ca7a-a16e-44bd-af55-d8babaa94e01" (UID: "1017ca7a-a16e-44bd-af55-d8babaa94e01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.530676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.530649 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istiod-ca-cert" pod "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" (UID: "e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:31.530889 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.530866 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1017ca7a-a16e-44bd-af55-d8babaa94e01-istiod-ca-cert" pod "1017ca7a-a16e-44bd-af55-d8babaa94e01" (UID: "1017ca7a-a16e-44bd-af55-d8babaa94e01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:31.534582 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.534555 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-envoy" pod "a6f473a6-9517-4367-b935-294dbd28ceec" (UID: "a6f473a6-9517-4367-b935-294dbd28ceec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.563140 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.562619 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-certs" pod "a6f473a6-9517-4367-b935-294dbd28ceec" (UID: "a6f473a6-9517-4367-b935-294dbd28ceec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.564789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.564649 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:58:31.571550 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.571527 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.571640 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.571554 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m875z\" (UniqueName: \"kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-kube-api-access-m875z\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.571640 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.571571 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.571640 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.571584 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q5t4\" (UniqueName: \"kubernetes.io/projected/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-kube-api-access-9q5t4\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.571640 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.571599 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.571640 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.571612 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.571640 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.571624 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.571640 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.571636 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.571653 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1017ca7a-a16e-44bd-af55-d8babaa94e01-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.571678 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snm8f\" (UniqueName: \"kubernetes.io/projected/e9c9e5ed-908c-463d-8f4f-395a80ababdd-kube-api-access-snm8f\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.572084 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.572118 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.572134 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpr7l\" (UniqueName: \"kubernetes.io/projected/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f-kube-api-access-vpr7l\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.572149 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.572162 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94094c6-876a-4082-ad7d-f35bb1909f87-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.572180 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.572194 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.572211 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.572224 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.572239 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1017ca7a-a16e-44bd-af55-d8babaa94e01-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.572253 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.572253 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpzk6\" (UniqueName: \"kubernetes.io/projected/1017ca7a-a16e-44bd-af55-d8babaa94e01-kube-api-access-vpzk6\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.604771 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.604639 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-data" pod "a6f473a6-9517-4367-b935-294dbd28ceec" (UID: "a6f473a6-9517-4367-b935-294dbd28ceec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.615802 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.607585 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-kube-api-access-gplkm" pod "a6f473a6-9517-4367-b935-294dbd28ceec" (UID: "a6f473a6-9517-4367-b935-294dbd28ceec"). InnerVolumeSpecName "kube-api-access-gplkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:31.615802 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.608755 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-socket" pod "a6f473a6-9517-4367-b935-294dbd28ceec" (UID: "a6f473a6-9517-4367-b935-294dbd28ceec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.627954 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.626111 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-istio-token" pod "a6f473a6-9517-4367-b935-294dbd28ceec" (UID: "a6f473a6-9517-4367-b935-294dbd28ceec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:31.661598 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.661338 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-credential-socket" pod "a6f473a6-9517-4367-b935-294dbd28ceec" (UID: "a6f473a6-9517-4367-b935-294dbd28ceec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.673530 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.673497 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-envoy\") pod \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " Oct 30 23:58:31.673678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.673585 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-token\" (UniqueName: \"kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-token\") pod \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " Oct 30 23:58:31.673817 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.673798 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-podinfo\") pod \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " Oct 30 23:58:31.673999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.673942 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-certs\") pod \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " Oct 30 23:58:31.674058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.674035 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-data\") pod \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " Oct 30 23:58:31.674156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.674141 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istiod-ca-cert\") pod \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " Oct 30 23:58:31.674314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.674297 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-kube-api-access-nfswb\" (UniqueName: \"kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-kube-api-access-nfswb\") pod \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " Oct 30 23:58:31.674575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.674556 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-credential-socket\") pod \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " Oct 30 23:58:31.674812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.674794 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-socket\") pod \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\" (UID: \"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7\") " Oct 30 23:58:31.676111 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.676075 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.676266 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.676251 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.676395 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.676379 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gplkm\" (UniqueName: \"kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-kube-api-access-gplkm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.676526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.676514 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f473a6-9517-4367-b935-294dbd28ceec-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.676624 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.676613 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f473a6-9517-4367-b935-294dbd28ceec-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.686128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.686098 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6f473a6-9517-4367-b935-294dbd28ceec-istio-podinfo" pod "a6f473a6-9517-4367-b935-294dbd28ceec" (UID: "a6f473a6-9517-4367-b935-294dbd28ceec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:58:31.694898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.694868 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f473a6-9517-4367-b935-294dbd28ceec-istiod-ca-cert" pod "a6f473a6-9517-4367-b935-294dbd28ceec" (UID: "a6f473a6-9517-4367-b935-294dbd28ceec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:31.730501 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.730423 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-envoy" pod "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" (UID: "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.749948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.749903 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-token" pod "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" (UID: "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:31.752620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.752580 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-certs" pod "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" (UID: "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.781542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.771055 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:58:31.785586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.784287 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f473a6-9517-4367-b935-294dbd28ceec-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.785586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.784331 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.785586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.784347 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.785586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.784360 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.785586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.784373 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f473a6-9517-4367-b935-294dbd28ceec-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.794710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.794685 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-credential-socket" pod "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" (UID: "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.794710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.794682 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-socket" pod "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" (UID: "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.795693 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.795665 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-data" pod "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" (UID: "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:31.796853 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.796831 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-podinfo" pod "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" (UID: "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:58:31.796948 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.796929 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-kube-api-access-nfswb" pod "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" (UID: "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7"). InnerVolumeSpecName "kube-api-access-nfswb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:31.840646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.840613 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istiod-ca-cert" pod "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" (UID: "29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:31.876377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.875972 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:58:31.885296 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.885267 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-socket\") pod \"f179a0a0-c464-4a8b-b6cc-04a21610a832\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " Oct 30 23:58:31.885422 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.885406 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-kube-api-access-94sb9\" (UniqueName: \"kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-kube-api-access-94sb9\") pod \"f179a0a0-c464-4a8b-b6cc-04a21610a832\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " Oct 30 23:58:31.885499 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.885483 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-token\" (UniqueName: \"kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-token\") pod \"f179a0a0-c464-4a8b-b6cc-04a21610a832\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " Oct 30 23:58:31.885546 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.885532 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-envoy\") pod \"f179a0a0-c464-4a8b-b6cc-04a21610a832\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " Oct 30 23:58:31.885599 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.885563 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-certs\") pod \"f179a0a0-c464-4a8b-b6cc-04a21610a832\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " Oct 30 23:58:31.885655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.885608 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-podinfo\") pod \"f179a0a0-c464-4a8b-b6cc-04a21610a832\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " Oct 30 23:58:31.885655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.885642 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f179a0a0-c464-4a8b-b6cc-04a21610a832-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f179a0a0-c464-4a8b-b6cc-04a21610a832-istiod-ca-cert\") pod \"f179a0a0-c464-4a8b-b6cc-04a21610a832\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " Oct 30 23:58:31.885779 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.885761 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-data\") pod \"f179a0a0-c464-4a8b-b6cc-04a21610a832\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " Oct 30 23:58:31.886027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.886014 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-credential-socket\") pod \"f179a0a0-c464-4a8b-b6cc-04a21610a832\" (UID: \"f179a0a0-c464-4a8b-b6cc-04a21610a832\") " Oct 30 23:58:31.887221 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.887197 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.887265 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.887232 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.887312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.887267 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.887312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.887283 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.887374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.887315 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfswb\" (UniqueName: \"kubernetes.io/projected/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-kube-api-access-nfswb\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.887374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.887331 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:31.965379 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.965284 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-podinfo" pod "f179a0a0-c464-4a8b-b6cc-04a21610a832" (UID: "f179a0a0-c464-4a8b-b6cc-04a21610a832"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:58:31.988765 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.988741 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-envoy\") pod \"af49f007-5f21-43cc-b4f3-d78e31b45100\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " Oct 30 23:58:31.988894 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.988843 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-certs\") pod \"af49f007-5f21-43cc-b4f3-d78e31b45100\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " Oct 30 23:58:31.989017 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.989005 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-token\" (UniqueName: \"kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-token\") pod \"af49f007-5f21-43cc-b4f3-d78e31b45100\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " Oct 30 23:58:31.989129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.989117 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-data\") pod \"af49f007-5f21-43cc-b4f3-d78e31b45100\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " Oct 30 23:58:31.989166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.989148 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-kube-api-access-8r6db\" (UniqueName: \"kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-kube-api-access-8r6db\") pod \"af49f007-5f21-43cc-b4f3-d78e31b45100\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " Oct 30 23:58:31.989223 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.989213 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-podinfo\") pod \"af49f007-5f21-43cc-b4f3-d78e31b45100\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " Oct 30 23:58:31.989261 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.989234 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-credential-socket\") pod \"af49f007-5f21-43cc-b4f3-d78e31b45100\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " Oct 30 23:58:31.989341 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.989331 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af49f007-5f21-43cc-b4f3-d78e31b45100-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af49f007-5f21-43cc-b4f3-d78e31b45100-istiod-ca-cert\") pod \"af49f007-5f21-43cc-b4f3-d78e31b45100\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " Oct 30 23:58:31.989416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.989406 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-socket\") pod \"af49f007-5f21-43cc-b4f3-d78e31b45100\" (UID: \"af49f007-5f21-43cc-b4f3-d78e31b45100\") " Oct 30 23:58:31.990222 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:31.990205 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.001521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.001301 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-certs" pod "af49f007-5f21-43cc-b4f3-d78e31b45100" (UID: "af49f007-5f21-43cc-b4f3-d78e31b45100"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.002339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.001697 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-socket" pod "af49f007-5f21-43cc-b4f3-d78e31b45100" (UID: "af49f007-5f21-43cc-b4f3-d78e31b45100"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.002339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.001860 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-credential-socket" pod "af49f007-5f21-43cc-b4f3-d78e31b45100" (UID: "af49f007-5f21-43cc-b4f3-d78e31b45100"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.002339 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.001945 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-data" pod "af49f007-5f21-43cc-b4f3-d78e31b45100" (UID: "af49f007-5f21-43cc-b4f3-d78e31b45100"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.002556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.002475 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af49f007-5f21-43cc-b4f3-d78e31b45100-istiod-ca-cert" pod "af49f007-5f21-43cc-b4f3-d78e31b45100" (UID: "af49f007-5f21-43cc-b4f3-d78e31b45100"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:32.002919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.002731 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:58:32.004418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.004382 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-token" pod "af49f007-5f21-43cc-b4f3-d78e31b45100" (UID: "af49f007-5f21-43cc-b4f3-d78e31b45100"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:32.004556 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.004456 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-podinfo" pod "af49f007-5f21-43cc-b4f3-d78e31b45100" (UID: "af49f007-5f21-43cc-b4f3-d78e31b45100"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:58:32.005656 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.005636 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-kube-api-access-8r6db" pod "af49f007-5f21-43cc-b4f3-d78e31b45100" (UID: "af49f007-5f21-43cc-b4f3-d78e31b45100"). InnerVolumeSpecName "kube-api-access-8r6db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:32.005736 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.005656 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-envoy" pod "af49f007-5f21-43cc-b4f3-d78e31b45100" (UID: "af49f007-5f21-43cc-b4f3-d78e31b45100"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.015540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.015512 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-credential-socket" pod "f179a0a0-c464-4a8b-b6cc-04a21610a832" (UID: "f179a0a0-c464-4a8b-b6cc-04a21610a832"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.060394 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.060364 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:32.091551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.091522 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12dc177-11a1-40b0-9fdf-75993ab176e6-kube-api-access-xznl4\" (UniqueName: \"kubernetes.io/projected/b12dc177-11a1-40b0-9fdf-75993ab176e6-kube-api-access-xznl4\") pod \"b12dc177-11a1-40b0-9fdf-75993ab176e6\" (UID: \"b12dc177-11a1-40b0-9fdf-75993ab176e6\") " Oct 30 23:58:32.091836 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.091819 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume\") pod \"b12dc177-11a1-40b0-9fdf-75993ab176e6\" (UID: \"b12dc177-11a1-40b0-9fdf-75993ab176e6\") " Oct 30 23:58:32.092651 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.092619 7570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" podUID="20fc3e5f-362e-41f7-bb52-2b171d7af818" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:58:32.092754 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.092697 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.092754 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.092718 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.092754 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.092731 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.092754 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.092748 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.092908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.092761 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.092908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.092774 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.092908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.092788 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r6db\" (UniqueName: \"kubernetes.io/projected/af49f007-5f21-43cc-b4f3-d78e31b45100-kube-api-access-8r6db\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.092908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.092801 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af49f007-5f21-43cc-b4f3-d78e31b45100-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.092908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.092814 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49f007-5f21-43cc-b4f3-d78e31b45100-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.092908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.092828 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af49f007-5f21-43cc-b4f3-d78e31b45100-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.094086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.094066 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12dc177-11a1-40b0-9fdf-75993ab176e6-kube-api-access-xznl4" pod "b12dc177-11a1-40b0-9fdf-75993ab176e6" (UID: "b12dc177-11a1-40b0-9fdf-75993ab176e6"). InnerVolumeSpecName "kube-api-access-xznl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:32.119752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.119716 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume" pod "b12dc177-11a1-40b0-9fdf-75993ab176e6" (UID: "b12dc177-11a1-40b0-9fdf-75993ab176e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:32.147263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.146771 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-certs" pod "f179a0a0-c464-4a8b-b6cc-04a21610a832" (UID: "f179a0a0-c464-4a8b-b6cc-04a21610a832"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.147263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.147213 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-data" pod "f179a0a0-c464-4a8b-b6cc-04a21610a832" (UID: "f179a0a0-c464-4a8b-b6cc-04a21610a832"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.147792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.147584 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-socket" pod "f179a0a0-c464-4a8b-b6cc-04a21610a832" (UID: "f179a0a0-c464-4a8b-b6cc-04a21610a832"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.147792 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.147742 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f179a0a0-c464-4a8b-b6cc-04a21610a832-istiod-ca-cert" pod "f179a0a0-c464-4a8b-b6cc-04a21610a832" (UID: "f179a0a0-c464-4a8b-b6cc-04a21610a832"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:32.195140 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.195107 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12dc177-11a1-40b0-9fdf-75993ab176e6-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.195140 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.195135 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.195140 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.195144 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.195357 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.195153 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f179a0a0-c464-4a8b-b6cc-04a21610a832-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.195357 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.195163 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.195357 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.195172 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xznl4\" (UniqueName: \"kubernetes.io/projected/b12dc177-11a1-40b0-9fdf-75993ab176e6-kube-api-access-xznl4\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.203803 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.203773 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-envoy" pod "f179a0a0-c464-4a8b-b6cc-04a21610a832" (UID: "f179a0a0-c464-4a8b-b6cc-04a21610a832"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.203911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.203811 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-token" pod "f179a0a0-c464-4a8b-b6cc-04a21610a832" (UID: "f179a0a0-c464-4a8b-b6cc-04a21610a832"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:32.203911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.203846 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-kube-api-access-94sb9" pod "f179a0a0-c464-4a8b-b6cc-04a21610a832" (UID: "f179a0a0-c464-4a8b-b6cc-04a21610a832"). InnerVolumeSpecName "kube-api-access-94sb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:32.203911 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.203877 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:58:32.223907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.223834 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" event={"ID":"5244915c-9573-489c-b1f8-e22948899df3","Type":"ContainerDied","Data":"8a970351fdd6076d4c0a29a1c6188fc87d517621dee89321299f23405c519104"} Oct 30 23:58:32.223907 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.223889 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a970351fdd6076d4c0a29a1c6188fc87d517621dee89321299f23405c519104" Oct 30 23:58:32.225665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.225640 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" event={"ID":"12efb180-e881-40a3-9001-8a39a2ddd352","Type":"ContainerDied","Data":"e31e4adbff9fdede5fa5e5267e5d0855347ca5ab40d7269790fdb9af9b1da01d"} Oct 30 23:58:32.225772 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.225683 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e31e4adbff9fdede5fa5e5267e5d0855347ca5ab40d7269790fdb9af9b1da01d" Oct 30 23:58:32.227200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.227184 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" event={"ID":"b94094c6-876a-4082-ad7d-f35bb1909f87","Type":"ContainerDied","Data":"5307084d1930531fb5103a70df19b2a4a36153c9a10084188240540440dd538c"} Oct 30 23:58:32.227276 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.227188 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24" Oct 30 23:58:32.228891 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.228873 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" Oct 30 23:58:32.228891 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.228880 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9" event={"ID":"29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7","Type":"ContainerDied","Data":"78a783eba81c05eca8bc4b8264e20b1cc37a37d9c8b5ac15d7b0ba42c99568c5"} Oct 30 23:58:32.231643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.231608 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" event={"ID":"d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e","Type":"ContainerStarted","Data":"a1db67ab646c62b10588afc694c789abb45b9bc3a9638785bcd6a30c4655de0b"} Oct 30 23:58:32.233519 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.233501 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" Oct 30 23:58:32.233603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.233518 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl" event={"ID":"a6f473a6-9517-4367-b935-294dbd28ceec","Type":"ContainerDied","Data":"fbd4314b7b3d8c3cc09a22d3a942278f369347bbfa7a0333bac954721569d92b"} Oct 30 23:58:32.235835 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.235815 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" event={"ID":"06e6c870-08ba-4b3c-a26d-45397d5ca730","Type":"ContainerStarted","Data":"aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b"} Oct 30 23:58:32.235992 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.235942 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:58:32.238579 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.238191 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" event={"ID":"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5","Type":"ContainerStarted","Data":"941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad"} Oct 30 23:58:32.238579 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.238529 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:58:32.242115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.242023 7570 scope.go:122] "RemoveContainer" containerID="437b364aa15062825ed4d26c6d17ec8b15ba4dd0916980db86c3f0f71ca026e9" Oct 30 23:58:32.242194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.242115 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x6pgq" event={"ID":"2b47b3cc-9735-45f1-8d87-7edd6ef95dd8","Type":"ContainerStarted","Data":"42e3033f32b7ec052331ce65ebcc58960b478877c556dd2b6a618f83e13ed257"} Oct 30 23:58:32.245579 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.245556 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" event={"ID":"2533ebe5-857f-4255-8c85-5a526fe1062b","Type":"ContainerStarted","Data":"a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571"} Oct 30 23:58:32.245740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.245641 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:58:32.247502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.247438 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" event={"ID":"a74a48f3-2b68-424b-9489-33e22dab2f2f","Type":"ContainerDied","Data":"a043f90e7035a4e386bc15823e6385c6a5447d6ef823be9492939d680a4f67cc"} Oct 30 23:58:32.247598 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.247499 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a043f90e7035a4e386bc15823e6385c6a5447d6ef823be9492939d680a4f67cc" Oct 30 23:58:32.249203 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.249183 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" event={"ID":"aa08b804-5246-4fb8-a68c-cf7af9ba4a54","Type":"ContainerDied","Data":"292e48f0a27d0eb592da0c973a7bf973bbc8b4305edc93f0165f8f89486979b2"} Oct 30 23:58:32.249285 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.249227 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="292e48f0a27d0eb592da0c973a7bf973bbc8b4305edc93f0165f8f89486979b2" Oct 30 23:58:32.250478 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.250443 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" event={"ID":"fb14f0ca-115b-4fe6-afcd-6338df872573","Type":"ContainerDied","Data":"675e05d71c07535fcc78e3e3b109c12e74b9a8ac4072bf725391dbe1e0111ac9"} Oct 30 23:58:32.250576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.250524 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="675e05d71c07535fcc78e3e3b109c12e74b9a8ac4072bf725391dbe1e0111ac9" Oct 30 23:58:32.252053 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.252033 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" event={"ID":"d8235a15-af1b-4332-ac83-1aef5bfb1286","Type":"ContainerDied","Data":"054e2ef4159b116b3a8a75ec865d77186663a5477768c85d5fe580610f584c79"} Oct 30 23:58:32.252127 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.252068 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="054e2ef4159b116b3a8a75ec865d77186663a5477768c85d5fe580610f584c79" Oct 30 23:58:32.254226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.254204 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" event={"ID":"c15181ee-3e9a-40c9-a029-9ee504a971fa","Type":"ContainerStarted","Data":"225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72"} Oct 30 23:58:32.254391 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.254377 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:58:32.255749 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.255727 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" event={"ID":"af49f007-5f21-43cc-b4f3-d78e31b45100","Type":"ContainerDied","Data":"cfbb610c13b3992de772f7c9b258ac48d3e90f25023b56ae157ae0c0d3dd5229"} Oct 30 23:58:32.255827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.255729 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk" Oct 30 23:58:32.257933 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.257878 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-4blbf" event={"ID":"1793bf9b-92db-4dae-bfd6-f86e5b076739","Type":"ContainerStarted","Data":"9d3c40cb76ab99f305a5e9e4944e41de5a9ec5577e76082359fb79c698f26e78"} Oct 30 23:58:32.258041 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.258027 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:58:32.260296 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.260276 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" event={"ID":"4af4419c-fffd-463e-9577-94bad299abc6","Type":"ContainerStarted","Data":"5cbc0db062b1b3fe35cd3806d520010a27549f62c95551106e51ac669550b876"} Oct 30 23:58:32.260810 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.260763 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:58:32.263069 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.263048 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" event={"ID":"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903","Type":"ContainerStarted","Data":"12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61"} Oct 30 23:58:32.263190 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.263174 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:58:32.264747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.264723 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" Oct 30 23:58:32.264839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.264724 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx" event={"ID":"f68436c1-d900-4063-b853-cb9be78ceae6","Type":"ContainerDied","Data":"ac32738d430a6d0e77a64d5fd40c9f6152a014dfe151d4b64388ce9511468003"} Oct 30 23:58:32.266827 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.266805 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" event={"ID":"6b3d259c-4f6c-4033-b6dd-1059616fe3b4","Type":"ContainerStarted","Data":"b68cf4f3a12e3c954ae3cd6c6dc399443a0cfd9ec46ace8c996500adcef15c2c"} Oct 30 23:58:32.266931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.266909 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:58:32.268931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.268898 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" event={"ID":"eedd4bdb-1495-47e6-884d-5ce26a05e72b","Type":"ContainerStarted","Data":"2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d"} Oct 30 23:58:32.269098 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.268993 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:58:32.270388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.270353 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" Oct 30 23:58:32.270443 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.270391 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf" event={"ID":"f179a0a0-c464-4a8b-b6cc-04a21610a832","Type":"ContainerDied","Data":"4b453b82afffbfbaa0b1404213c2ba27f6c3443fb37a1de5aad05d9c30a3bc35"} Oct 30 23:58:32.272088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.272030 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" Oct 30 23:58:32.272088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.272037 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2" event={"ID":"b12dc177-11a1-40b0-9fdf-75993ab176e6","Type":"ContainerDied","Data":"a93e97b08d4fc30ddc8c46d8359c87022076209830fa3c0fa007e37b7a8f89c5"} Oct 30 23:58:32.273693 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.273674 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" event={"ID":"c9e446a0-c6da-40ab-aa1f-f3756e39619a","Type":"ContainerDied","Data":"b22fff677a45cd41ef912f36f018f4e9da388b1fc09af78c9bd149702466e162"} Oct 30 23:58:32.273757 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.273714 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b22fff677a45cd41ef912f36f018f4e9da388b1fc09af78c9bd149702466e162" Oct 30 23:58:32.296377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.296349 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-socket\") pod \"f68436c1-d900-4063-b853-cb9be78ceae6\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " Oct 30 23:58:32.296503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.296395 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f68436c1-d900-4063-b853-cb9be78ceae6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f68436c1-d900-4063-b853-cb9be78ceae6-istio-podinfo\") pod \"f68436c1-d900-4063-b853-cb9be78ceae6\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " Oct 30 23:58:32.296503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.296437 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-istio-token\" (UniqueName: \"kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-istio-token\") pod \"f68436c1-d900-4063-b853-cb9be78ceae6\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " Oct 30 23:58:32.296618 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.296514 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f68436c1-d900-4063-b853-cb9be78ceae6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f68436c1-d900-4063-b853-cb9be78ceae6-istiod-ca-cert\") pod \"f68436c1-d900-4063-b853-cb9be78ceae6\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " Oct 30 23:58:32.296618 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.296538 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-certs\") pod \"f68436c1-d900-4063-b853-cb9be78ceae6\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " Oct 30 23:58:32.296706 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.296621 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-envoy\") pod \"f68436c1-d900-4063-b853-cb9be78ceae6\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " Oct 30 23:58:32.296706 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.296656 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-credential-socket\") pod \"f68436c1-d900-4063-b853-cb9be78ceae6\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " Oct 30 23:58:32.296806 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.296790 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-data\") pod \"f68436c1-d900-4063-b853-cb9be78ceae6\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " Oct 30 23:58:32.296932 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.296916 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-kube-api-access-kckmj\" (UniqueName: \"kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-kube-api-access-kckmj\") pod \"f68436c1-d900-4063-b853-cb9be78ceae6\" (UID: \"f68436c1-d900-4063-b853-cb9be78ceae6\") " Oct 30 23:58:32.297813 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.297792 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94sb9\" (UniqueName: \"kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-kube-api-access-94sb9\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.297870 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.297824 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.297870 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.297839 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f179a0a0-c464-4a8b-b6cc-04a21610a832-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.335068 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.335043 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-credential-socket" pod "f68436c1-d900-4063-b853-cb9be78ceae6" (UID: "f68436c1-d900-4063-b853-cb9be78ceae6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.335429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.335402 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-data" pod "f68436c1-d900-4063-b853-cb9be78ceae6" (UID: "f68436c1-d900-4063-b853-cb9be78ceae6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.335725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.335682 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68436c1-d900-4063-b853-cb9be78ceae6-istiod-ca-cert" pod "f68436c1-d900-4063-b853-cb9be78ceae6" (UID: "f68436c1-d900-4063-b853-cb9be78ceae6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:32.336238 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.336200 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-certs" pod "f68436c1-d900-4063-b853-cb9be78ceae6" (UID: "f68436c1-d900-4063-b853-cb9be78ceae6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.337327 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.336635 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-socket" pod "f68436c1-d900-4063-b853-cb9be78ceae6" (UID: "f68436c1-d900-4063-b853-cb9be78ceae6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.338298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.338260 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-kube-api-access-kckmj" pod "f68436c1-d900-4063-b853-cb9be78ceae6" (UID: "f68436c1-d900-4063-b853-cb9be78ceae6"). InnerVolumeSpecName "kube-api-access-kckmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:32.354772 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.354751 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f68436c1-d900-4063-b853-cb9be78ceae6-istio-podinfo" pod "f68436c1-d900-4063-b853-cb9be78ceae6" (UID: "f68436c1-d900-4063-b853-cb9be78ceae6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:58:32.371833 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.371811 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-envoy" pod "f68436c1-d900-4063-b853-cb9be78ceae6" (UID: "f68436c1-d900-4063-b853-cb9be78ceae6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.371975 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.371948 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-istio-token" pod "f68436c1-d900-4063-b853-cb9be78ceae6" (UID: "f68436c1-d900-4063-b853-cb9be78ceae6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:32.374079 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.374057 7570 scope.go:122] "RemoveContainer" containerID="0bde901d48fa5140c259012bee5dd70f0ef26c4ba298b70a93e1e9c3ff2d7acd" Oct 30 23:58:32.397755 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.397730 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:58:32.401100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.401079 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.401100 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.401102 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.401218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.401111 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.401218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.401120 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kckmj\" (UniqueName: \"kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-kube-api-access-kckmj\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.401218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.401130 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.401218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.401139 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f68436c1-d900-4063-b853-cb9be78ceae6-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.401218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.401148 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68436c1-d900-4063-b853-cb9be78ceae6-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.401218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.401156 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f68436c1-d900-4063-b853-cb9be78ceae6-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.401218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.401164 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68436c1-d900-4063-b853-cb9be78ceae6-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.498766 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.498628 7570 scope.go:122] "RemoveContainer" containerID="7b62593c2ba5d2d1053164bdc55539dce34edcf66c3bf756b2febfddb756f124" Oct 30 23:58:32.502507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.501977 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-socket\") pod \"d8235a15-af1b-4332-ac83-1aef5bfb1286\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " Oct 30 23:58:32.502507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.502039 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-credential-socket\") pod \"d8235a15-af1b-4332-ac83-1aef5bfb1286\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " Oct 30 23:58:32.502507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.502155 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-data\") pod \"d8235a15-af1b-4332-ac83-1aef5bfb1286\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " Oct 30 23:58:32.502507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.502227 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-certs\") pod \"d8235a15-af1b-4332-ac83-1aef5bfb1286\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " Oct 30 23:58:32.502507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.502282 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-podinfo\") pod \"d8235a15-af1b-4332-ac83-1aef5bfb1286\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " Oct 30 23:58:32.504212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.502830 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8235a15-af1b-4332-ac83-1aef5bfb1286-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8235a15-af1b-4332-ac83-1aef5bfb1286-istiod-ca-cert\") pod \"d8235a15-af1b-4332-ac83-1aef5bfb1286\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " Oct 30 23:58:32.504212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.502894 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-kube-api-access-82qgh\" (UniqueName: \"kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-kube-api-access-82qgh\") pod \"d8235a15-af1b-4332-ac83-1aef5bfb1286\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " Oct 30 23:58:32.504212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.502940 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-token\" (UniqueName: \"kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-token\") pod \"d8235a15-af1b-4332-ac83-1aef5bfb1286\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " Oct 30 23:58:32.504212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.502966 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-envoy\") pod \"d8235a15-af1b-4332-ac83-1aef5bfb1286\" (UID: \"d8235a15-af1b-4332-ac83-1aef5bfb1286\") " Oct 30 23:58:32.578090 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.578059 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8235a15-af1b-4332-ac83-1aef5bfb1286-istiod-ca-cert" pod "d8235a15-af1b-4332-ac83-1aef5bfb1286" (UID: "d8235a15-af1b-4332-ac83-1aef5bfb1286"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:32.579094 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.578745 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:58:32.613630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.613595 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8235a15-af1b-4332-ac83-1aef5bfb1286-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.698753 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.698649 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-envoy" pod "d8235a15-af1b-4332-ac83-1aef5bfb1286" (UID: "d8235a15-af1b-4332-ac83-1aef5bfb1286"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.699077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.699034 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-socket" pod "d8235a15-af1b-4332-ac83-1aef5bfb1286" (UID: "d8235a15-af1b-4332-ac83-1aef5bfb1286"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.699233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.699167 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-credential-socket" pod "d8235a15-af1b-4332-ac83-1aef5bfb1286" (UID: "d8235a15-af1b-4332-ac83-1aef5bfb1286"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.699233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.699178 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-certs" pod "d8235a15-af1b-4332-ac83-1aef5bfb1286" (UID: "d8235a15-af1b-4332-ac83-1aef5bfb1286"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.714644 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.714614 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-kube-api-access-pv58n\" (UniqueName: \"kubernetes.io/projected/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-kube-api-access-pv58n\") pod \"aa08b804-5246-4fb8-a68c-cf7af9ba4a54\" (UID: \"aa08b804-5246-4fb8-a68c-cf7af9ba4a54\") " Oct 30 23:58:32.714805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.714790 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume\") pod \"aa08b804-5246-4fb8-a68c-cf7af9ba4a54\" (UID: \"aa08b804-5246-4fb8-a68c-cf7af9ba4a54\") " Oct 30 23:58:32.715422 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.715401 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.715596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.715426 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.715596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.715439 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.715596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.715452 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.798088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.794855 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-token" pod "d8235a15-af1b-4332-ac83-1aef5bfb1286" (UID: "d8235a15-af1b-4332-ac83-1aef5bfb1286"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:32.798357 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.798324 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-kube-api-access-82qgh" pod "d8235a15-af1b-4332-ac83-1aef5bfb1286" (UID: "d8235a15-af1b-4332-ac83-1aef5bfb1286"). InnerVolumeSpecName "kube-api-access-82qgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:32.798595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.798564 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-podinfo" pod "d8235a15-af1b-4332-ac83-1aef5bfb1286" (UID: "d8235a15-af1b-4332-ac83-1aef5bfb1286"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:58:32.799267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.799224 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-data" pod "d8235a15-af1b-4332-ac83-1aef5bfb1286" (UID: "d8235a15-af1b-4332-ac83-1aef5bfb1286"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:32.799897 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.799867 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume" pod "aa08b804-5246-4fb8-a68c-cf7af9ba4a54" (UID: "aa08b804-5246-4fb8-a68c-cf7af9ba4a54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:32.802354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.802320 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-kube-api-access-pv58n" pod "aa08b804-5246-4fb8-a68c-cf7af9ba4a54" (UID: "aa08b804-5246-4fb8-a68c-cf7af9ba4a54"). InnerVolumeSpecName "kube-api-access-pv58n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:32.818378 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.818344 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.818378 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.818381 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.818585 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.818398 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv58n\" (UniqueName: \"kubernetes.io/projected/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-kube-api-access-pv58n\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.818585 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.818416 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qgh\" (UniqueName: \"kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-kube-api-access-82qgh\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.818585 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.818429 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8235a15-af1b-4332-ac83-1aef5bfb1286-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.818585 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.818437 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa08b804-5246-4fb8-a68c-cf7af9ba4a54-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.827982 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.827952 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:58:32.881879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.881851 7570 scope.go:122] "RemoveContainer" containerID="0170737f67453b691218a3069bf111fefb4d960cddcbbb0718f56f6f8b6dd3b2" Oct 30 23:58:32.919633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.919605 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb14f0ca-115b-4fe6-afcd-6338df872573-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb14f0ca-115b-4fe6-afcd-6338df872573-plugins-config-volume\") pod \"fb14f0ca-115b-4fe6-afcd-6338df872573\" (UID: \"fb14f0ca-115b-4fe6-afcd-6338df872573\") " Oct 30 23:58:32.919753 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.919707 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb14f0ca-115b-4fe6-afcd-6338df872573-kube-api-access-fx4dn\" (UniqueName: \"kubernetes.io/projected/fb14f0ca-115b-4fe6-afcd-6338df872573-kube-api-access-fx4dn\") pod \"fb14f0ca-115b-4fe6-afcd-6338df872573\" (UID: \"fb14f0ca-115b-4fe6-afcd-6338df872573\") " Oct 30 23:58:32.920086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.920060 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb14f0ca-115b-4fe6-afcd-6338df872573-plugins-config-volume" pod "fb14f0ca-115b-4fe6-afcd-6338df872573" (UID: "fb14f0ca-115b-4fe6-afcd-6338df872573"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:32.920541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.920520 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb14f0ca-115b-4fe6-afcd-6338df872573-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:32.922007 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:32.921987 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb14f0ca-115b-4fe6-afcd-6338df872573-kube-api-access-fx4dn" pod "fb14f0ca-115b-4fe6-afcd-6338df872573" (UID: "fb14f0ca-115b-4fe6-afcd-6338df872573"). InnerVolumeSpecName "kube-api-access-fx4dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:33.014123 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.014098 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:58:33.022590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.022560 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx4dn\" (UniqueName: \"kubernetes.io/projected/fb14f0ca-115b-4fe6-afcd-6338df872573-kube-api-access-fx4dn\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.119963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.119939 7570 scope.go:122] "RemoveContainer" containerID="afc2d23e0695cedbf256b4a7750b95ea9fcba96be1a1b86cbbbaa52d2c4a5cf5" Oct 30 23:58:33.124505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.124437 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74a48f3-2b68-424b-9489-33e22dab2f2f-kube-api-access-zms24\" (UniqueName: \"kubernetes.io/projected/a74a48f3-2b68-424b-9489-33e22dab2f2f-kube-api-access-zms24\") pod \"a74a48f3-2b68-424b-9489-33e22dab2f2f\" (UID: \"a74a48f3-2b68-424b-9489-33e22dab2f2f\") " Oct 30 23:58:33.124628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.124606 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74a48f3-2b68-424b-9489-33e22dab2f2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74a48f3-2b68-424b-9489-33e22dab2f2f-plugins-config-volume\") pod \"a74a48f3-2b68-424b-9489-33e22dab2f2f\" (UID: \"a74a48f3-2b68-424b-9489-33e22dab2f2f\") " Oct 30 23:58:33.125349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.125320 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74a48f3-2b68-424b-9489-33e22dab2f2f-plugins-config-volume" pod "a74a48f3-2b68-424b-9489-33e22dab2f2f" (UID: "a74a48f3-2b68-424b-9489-33e22dab2f2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:33.126022 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.125709 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74a48f3-2b68-424b-9489-33e22dab2f2f-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.127116 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.127094 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74a48f3-2b68-424b-9489-33e22dab2f2f-kube-api-access-zms24" pod "a74a48f3-2b68-424b-9489-33e22dab2f2f" (UID: "a74a48f3-2b68-424b-9489-33e22dab2f2f"). InnerVolumeSpecName "kube-api-access-zms24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:33.157044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.156957 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:58:33.227668 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.227637 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zms24\" (UniqueName: \"kubernetes.io/projected/a74a48f3-2b68-424b-9489-33e22dab2f2f-kube-api-access-zms24\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.256515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.256440 7570 scope.go:122] "RemoveContainer" containerID="a484bb709e0b21eb10b3c20ec1ffbc432be3950bc50ca72732a27baf0121f5d4" Oct 30 23:58:33.290808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.290786 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll" Oct 30 23:58:33.291001 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.290984 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk" Oct 30 23:58:33.292399 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.292374 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc" Oct 30 23:58:33.292655 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.292638 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g" Oct 30 23:58:33.292762 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.292750 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8" Oct 30 23:58:33.324762 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.324741 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:58:33.328215 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.328194 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12efb180-e881-40a3-9001-8a39a2ddd352-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12efb180-e881-40a3-9001-8a39a2ddd352-plugins-config-volume\") pod \"12efb180-e881-40a3-9001-8a39a2ddd352\" (UID: \"12efb180-e881-40a3-9001-8a39a2ddd352\") " Oct 30 23:58:33.328412 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.328383 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12efb180-e881-40a3-9001-8a39a2ddd352-kube-api-access-cjznr\" (UniqueName: \"kubernetes.io/projected/12efb180-e881-40a3-9001-8a39a2ddd352-kube-api-access-cjznr\") pod \"12efb180-e881-40a3-9001-8a39a2ddd352\" (UID: \"12efb180-e881-40a3-9001-8a39a2ddd352\") " Oct 30 23:58:33.328620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.328601 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12efb180-e881-40a3-9001-8a39a2ddd352-plugins-config-volume" pod "12efb180-e881-40a3-9001-8a39a2ddd352" (UID: "12efb180-e881-40a3-9001-8a39a2ddd352"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:33.329242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.329162 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12efb180-e881-40a3-9001-8a39a2ddd352-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.363665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.363625 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12efb180-e881-40a3-9001-8a39a2ddd352-kube-api-access-cjznr" pod "12efb180-e881-40a3-9001-8a39a2ddd352" (UID: "12efb180-e881-40a3-9001-8a39a2ddd352"). InnerVolumeSpecName "kube-api-access-cjznr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:33.364229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.364206 7570 scope.go:122] "RemoveContainer" containerID="ec286697b6cc7b25e983764c1e1a5728facf90554c840204c0c4c9e59148fe30" Oct 30 23:58:33.430230 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.430200 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-kube-api-access-tbfwt\" (UniqueName: \"kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-kube-api-access-tbfwt\") pod \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " Oct 30 23:58:33.430349 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.430313 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-data\") pod \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " Oct 30 23:58:33.430404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.430357 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istiod-ca-cert\") pod \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " Oct 30 23:58:33.430404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.430398 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-token\" (UniqueName: \"kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-token\") pod \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " Oct 30 23:58:33.430544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.430527 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-certs\") pod \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " Oct 30 23:58:33.430625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.430612 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-credential-socket\") pod \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " Oct 30 23:58:33.430720 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.430707 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-envoy\") pod \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " Oct 30 23:58:33.430774 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.430751 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-socket\") pod \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " Oct 30 23:58:33.430897 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.430885 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-podinfo\") pod \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\" (UID: \"c9e446a0-c6da-40ab-aa1f-f3756e39619a\") " Oct 30 23:58:33.433247 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.433224 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjznr\" (UniqueName: \"kubernetes.io/projected/12efb180-e881-40a3-9001-8a39a2ddd352-kube-api-access-cjznr\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.496925 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.496893 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:58:33.596784 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.596744 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-certs" pod "c9e446a0-c6da-40ab-aa1f-f3756e39619a" (UID: "c9e446a0-c6da-40ab-aa1f-f3756e39619a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:33.596938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.596783 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-socket" pod "c9e446a0-c6da-40ab-aa1f-f3756e39619a" (UID: "c9e446a0-c6da-40ab-aa1f-f3756e39619a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:33.596938 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.596853 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-envoy" pod "c9e446a0-c6da-40ab-aa1f-f3756e39619a" (UID: "c9e446a0-c6da-40ab-aa1f-f3756e39619a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:33.597213 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.597140 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-podinfo" pod "c9e446a0-c6da-40ab-aa1f-f3756e39619a" (UID: "c9e446a0-c6da-40ab-aa1f-f3756e39619a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:58:33.597330 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.597215 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-kube-api-access-tbfwt" pod "c9e446a0-c6da-40ab-aa1f-f3756e39619a" (UID: "c9e446a0-c6da-40ab-aa1f-f3756e39619a"). InnerVolumeSpecName "kube-api-access-tbfwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:33.597386 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.597352 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istiod-ca-cert" pod "c9e446a0-c6da-40ab-aa1f-f3756e39619a" (UID: "c9e446a0-c6da-40ab-aa1f-f3756e39619a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:33.597869 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.597842 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-data" pod "c9e446a0-c6da-40ab-aa1f-f3756e39619a" (UID: "c9e446a0-c6da-40ab-aa1f-f3756e39619a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:33.597975 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.597872 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-credential-socket" pod "c9e446a0-c6da-40ab-aa1f-f3756e39619a" (UID: "c9e446a0-c6da-40ab-aa1f-f3756e39619a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:33.600507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.600439 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-token" pod "c9e446a0-c6da-40ab-aa1f-f3756e39619a" (UID: "c9e446a0-c6da-40ab-aa1f-f3756e39619a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:33.616978 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.615746 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght"] Oct 30 23:58:33.626106 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.626083 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82d449-5c9fb54fbc-gvght"] Oct 30 23:58:33.642633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.637343 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5244915c-9573-489c-b1f8-e22948899df3-kube-api-access-6df9z\" (UniqueName: \"kubernetes.io/projected/5244915c-9573-489c-b1f8-e22948899df3-kube-api-access-6df9z\") pod \"5244915c-9573-489c-b1f8-e22948899df3\" (UID: \"5244915c-9573-489c-b1f8-e22948899df3\") " Oct 30 23:58:33.642633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.637425 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5244915c-9573-489c-b1f8-e22948899df3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5244915c-9573-489c-b1f8-e22948899df3-plugins-config-volume\") pod \"5244915c-9573-489c-b1f8-e22948899df3\" (UID: \"5244915c-9573-489c-b1f8-e22948899df3\") " Oct 30 23:58:33.642633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.638493 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.642633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.638518 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.642633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.638533 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.642633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.638548 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbfwt\" (UniqueName: \"kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-kube-api-access-tbfwt\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.642633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.638561 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.642633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.638574 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.642633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.638588 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e446a0-c6da-40ab-aa1f-f3756e39619a-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.642633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.638601 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.642633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.638617 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e446a0-c6da-40ab-aa1f-f3756e39619a-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.642633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.641540 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c"] Oct 30 23:58:33.647527 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.647061 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-df3b5f-7f86b94c59-l9x4c"] Oct 30 23:58:33.666445 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.666369 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr"] Oct 30 23:58:33.672685 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.672660 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f026c1-openshift-default-5bf6b99cc5-8wgpr"] Oct 30 23:58:33.680526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.680445 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5244915c-9573-489c-b1f8-e22948899df3-plugins-config-volume" pod "5244915c-9573-489c-b1f8-e22948899df3" (UID: "5244915c-9573-489c-b1f8-e22948899df3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:33.683408 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.683385 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5244915c-9573-489c-b1f8-e22948899df3-kube-api-access-6df9z" pod "5244915c-9573-489c-b1f8-e22948899df3" (UID: "5244915c-9573-489c-b1f8-e22948899df3"). InnerVolumeSpecName "kube-api-access-6df9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:33.711089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.711049 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6"] Oct 30 23:58:33.719144 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.719119 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-thnp6"] Oct 30 23:58:33.740011 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.739984 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6df9z\" (UniqueName: \"kubernetes.io/projected/5244915c-9573-489c-b1f8-e22948899df3-kube-api-access-6df9z\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.740137 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.740014 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5244915c-9573-489c-b1f8-e22948899df3-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:33.778526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.778455 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g"] Oct 30 23:58:33.786960 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.786933 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82d449-9679f64c9-zgj7g"] Oct 30 23:58:33.830812 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.830780 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9"] Oct 30 23:58:33.870650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.870618 7570 scope.go:122] "RemoveContainer" containerID="8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50" Oct 30 23:58:33.876446 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:33.876401 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50\": container with ID starting with 8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50 not found: ID does not exist" containerID="8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50" Oct 30 23:58:33.876632 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.876601 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50"} err="failed to get container status \"8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50\": rpc error: code = NotFound desc = could not find container \"8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50\": container with ID starting with 8979fcb954d18ddae2483c1c01201fcc3320de850e369ac90a2156025457fc50 not found: ID does not exist" Oct 30 23:58:33.876632 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.876631 7570 scope.go:122] "RemoveContainer" containerID="6e10c215485e3d6ccb26655f26754e13b3bcda5a3f83836c46072df0e29cc25a" Oct 30 23:58:33.876927 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.876912 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-vcpd9"] Oct 30 23:58:33.920242 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.920200 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll"] Oct 30 23:58:33.927594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.927571 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-23f75e-55b569f787-tqdll"] Oct 30 23:58:33.930347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.930330 7570 scope.go:122] "RemoveContainer" containerID="c30617db6774365f2605f645b02abd554fc4a67999922b27bc2763f1ad73c11a" Oct 30 23:58:34.001184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:33.998070 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc"] Oct 30 23:58:34.007887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.007265 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b0299-5869f47955-r42zc"] Oct 30 23:58:34.008328 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.008304 7570 scope.go:122] "RemoveContainer" containerID="29f5ecced49b3f5128454a5896504d057362df6f345169fd68da0bbb5fa23782" Oct 30 23:58:34.037898 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.037872 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24"] Oct 30 23:58:34.046757 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.046723 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-4wc24"] Oct 30 23:58:34.134497 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.132533 7570 scope.go:122] "RemoveContainer" containerID="a2397d1a972d4c1e20267b88f3965c58ed65da73260e8e63c61ae66c513cbdab" Oct 30 23:58:34.206902 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.206834 7570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:58:34.391013 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.389779 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log" Oct 30 23:58:34.391013 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.389983 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsv62" event={"ID":"29c3d1e4-09b5-4712-ad6d-9a32434def2d","Type":"ContainerStarted","Data":"c721051826e3cce4d11da842884942eebb8592a302fdb59a49c59e342ee1195d"} Oct 30 23:58:34.396630 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.396609 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws" Oct 30 23:58:34.396841 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.396641 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798" Oct 30 23:58:34.463432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.463361 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl"] Oct 30 23:58:34.485130 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.485100 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-bxdxl"] Oct 30 23:58:34.623722 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.623689 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" path="/var/lib/kubelet/pods/1017ca7a-a16e-44bd-af55-d8babaa94e01/volumes" Oct 30 23:58:34.625922 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.625900 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12efb180-e881-40a3-9001-8a39a2ddd352" path="/var/lib/kubelet/pods/12efb180-e881-40a3-9001-8a39a2ddd352/volumes" Oct 30 23:58:34.626574 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.626557 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" path="/var/lib/kubelet/pods/29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7/volumes" Oct 30 23:58:34.627122 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.627108 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" path="/var/lib/kubelet/pods/a6f473a6-9517-4367-b935-294dbd28ceec/volumes" Oct 30 23:58:34.628690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.628665 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74a48f3-2b68-424b-9489-33e22dab2f2f" path="/var/lib/kubelet/pods/a74a48f3-2b68-424b-9489-33e22dab2f2f/volumes" Oct 30 23:58:34.629089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.629074 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" path="/var/lib/kubelet/pods/aa08b804-5246-4fb8-a68c-cf7af9ba4a54/volumes" Oct 30 23:58:34.629628 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.629610 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" path="/var/lib/kubelet/pods/b4e6db40-bf7c-4440-bed0-8ec6e70cab0f/volumes" Oct 30 23:58:34.630305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.630291 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" path="/var/lib/kubelet/pods/b94094c6-876a-4082-ad7d-f35bb1909f87/volumes" Oct 30 23:58:34.631596 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.631577 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" path="/var/lib/kubelet/pods/e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984/volumes" Oct 30 23:58:34.632536 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.632521 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" path="/var/lib/kubelet/pods/e9c9e5ed-908c-463d-8f4f-395a80ababdd/volumes" Oct 30 23:58:34.632951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.632934 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk"] Oct 30 23:58:34.632994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.632971 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-df3b5f-openshift-default-5dfc87d949-dp2kk"] Oct 30 23:58:34.671604 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.671579 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx"] Oct 30 23:58:34.676491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.676455 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2b4009-openshift-default-554db6bc89-bv4tx"] Oct 30 23:58:34.807023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.806220 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8"] Oct 30 23:58:34.813446 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.813373 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-23f75e-575f9dbb8c-4q2n8"] Oct 30 23:58:34.907419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.907383 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk"] Oct 30 23:58:34.907729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.907668 7570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:58:34.207074515Z","UUID":"b87e19d2-f4d4-4a6f-885c-2e29c20a8d4e","Handler":null,"Name":"","Endpoint":""} Oct 30 23:58:34.909507 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.909431 7570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:58:34.909691 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.909671 7570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:58:34.922480 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.919565 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-82d449-openshift-default-6949995cc-pkpnk"] Oct 30 23:58:34.994896 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:34.994841 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf"] Oct 30 23:58:35.016744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.016713 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-7c5jf"] Oct 30 23:58:35.086760 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.086728 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnmqd" Oct 30 23:58:35.101333 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.101306 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:58:35.105142 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.105119 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2"] Oct 30 23:58:35.122620 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.122587 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2b4009-66ff9747c9-hl2z2"] Oct 30 23:58:35.144032 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.143994 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws"] Oct 30 23:58:35.157023 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.156995 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b0299-7966bc49c7-744ws"] Oct 30 23:58:35.198877 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.198848 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-podinfo\") pod \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " Oct 30 23:58:35.199044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.198952 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-credential-socket\") pod \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " Oct 30 23:58:35.199044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.199032 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-socket\") pod \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " Oct 30 23:58:35.199119 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.199054 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-token\" (UniqueName: \"kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-token\") pod \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " Oct 30 23:58:35.199119 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.199097 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-envoy\") pod \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " Oct 30 23:58:35.199171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.199138 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-certs\") pod \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " Oct 30 23:58:35.199199 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.199174 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-kube-api-access-kz2lm\" (UniqueName: \"kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-kube-api-access-kz2lm\") pod \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " Oct 30 23:58:35.199388 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.199376 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-data\") pod \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " Oct 30 23:58:35.199423 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.199398 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istiod-ca-cert\") pod \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\" (UID: \"4c7f3b9a-978c-49f2-a52a-5293dfb1924f\") " Oct 30 23:58:35.199931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.199906 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-credential-socket" pod "4c7f3b9a-978c-49f2-a52a-5293dfb1924f" (UID: "4c7f3b9a-978c-49f2-a52a-5293dfb1924f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:35.200690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.200312 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-socket" pod "4c7f3b9a-978c-49f2-a52a-5293dfb1924f" (UID: "4c7f3b9a-978c-49f2-a52a-5293dfb1924f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:35.201064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.200631 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:35.201064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.200812 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:35.201064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.200037 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-certs" pod "4c7f3b9a-978c-49f2-a52a-5293dfb1924f" (UID: "4c7f3b9a-978c-49f2-a52a-5293dfb1924f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:35.201064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.200853 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istiod-ca-cert" pod "4c7f3b9a-978c-49f2-a52a-5293dfb1924f" (UID: "4c7f3b9a-978c-49f2-a52a-5293dfb1924f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:35.201064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.201006 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-data" pod "4c7f3b9a-978c-49f2-a52a-5293dfb1924f" (UID: "4c7f3b9a-978c-49f2-a52a-5293dfb1924f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:35.202074 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.202045 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-podinfo" pod "4c7f3b9a-978c-49f2-a52a-5293dfb1924f" (UID: "4c7f3b9a-978c-49f2-a52a-5293dfb1924f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:58:35.202237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.202221 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-token" pod "4c7f3b9a-978c-49f2-a52a-5293dfb1924f" (UID: "4c7f3b9a-978c-49f2-a52a-5293dfb1924f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:35.202300 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.202251 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-kube-api-access-kz2lm" pod "4c7f3b9a-978c-49f2-a52a-5293dfb1924f" (UID: "4c7f3b9a-978c-49f2-a52a-5293dfb1924f"). InnerVolumeSpecName "kube-api-access-kz2lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:35.202778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.202755 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-envoy" pod "4c7f3b9a-978c-49f2-a52a-5293dfb1924f" (UID: "4c7f3b9a-978c-49f2-a52a-5293dfb1924f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:35.211694 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.211662 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798"] Oct 30 23:58:35.218740 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.218718 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-qr798"] Oct 30 23:58:35.302381 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.302348 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:35.302381 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.302380 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:35.302587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.302392 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:35.302587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.302400 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:35.302587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.302409 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:35.302587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.302418 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:35.302587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.302429 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz2lm\" (UniqueName: \"kubernetes.io/projected/4c7f3b9a-978c-49f2-a52a-5293dfb1924f-kube-api-access-kz2lm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:35.394745 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.392046 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5n89j" Oct 30 23:58:35.409027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.408996 7570 generic.go:356] "Generic (PLEG): container finished" podID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerID="041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931" exitCode=0 Oct 30 23:58:35.409172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.409082 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" Oct 30 23:58:35.409172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.409100 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" event={"ID":"4c7f3b9a-978c-49f2-a52a-5293dfb1924f","Type":"ContainerDied","Data":"041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931"} Oct 30 23:58:35.409305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.409179 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj" event={"ID":"4c7f3b9a-978c-49f2-a52a-5293dfb1924f","Type":"ContainerDied","Data":"f4ffedd06bba0b89cf93f59d2c4d3f40407db636de38c9fdaa74b9a6589ac587"} Oct 30 23:58:35.409305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.409212 7570 scope.go:122] "RemoveContainer" containerID="041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931" Oct 30 23:58:35.426571 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.426527 7570 scope.go:122] "RemoveContainer" containerID="041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931" Oct 30 23:58:35.426961 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:35.426940 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931\": container with ID starting with 041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931 not found: ID does not exist" containerID="041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931" Oct 30 23:58:35.427048 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.426970 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931"} err="failed to get container status \"041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931\": rpc error: code = NotFound desc = could not find container \"041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931\": container with ID starting with 041375e8665d0b610ad8e89d3d93c9f9700c2bd637136824c3e6d91cbab81931 not found: ID does not exist" Oct 30 23:58:35.471065 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.471033 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj"] Oct 30 23:58:35.479109 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:35.479085 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-4xnvj"] Oct 30 23:58:36.608207 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:36.608163 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" path="/var/lib/kubelet/pods/4c7f3b9a-978c-49f2-a52a-5293dfb1924f/volumes" Oct 30 23:58:36.608781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:36.608764 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5244915c-9573-489c-b1f8-e22948899df3" path="/var/lib/kubelet/pods/5244915c-9573-489c-b1f8-e22948899df3/volumes" Oct 30 23:58:36.609258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:36.609240 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" path="/var/lib/kubelet/pods/af49f007-5f21-43cc-b4f3-d78e31b45100/volumes" Oct 30 23:58:36.610012 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:36.609990 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12dc177-11a1-40b0-9fdf-75993ab176e6" path="/var/lib/kubelet/pods/b12dc177-11a1-40b0-9fdf-75993ab176e6/volumes" Oct 30 23:58:36.610999 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:36.610979 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" path="/var/lib/kubelet/pods/c9e446a0-c6da-40ab-aa1f-f3756e39619a/volumes" Oct 30 23:58:36.611678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:36.611659 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8235a15-af1b-4332-ac83-1aef5bfb1286" path="/var/lib/kubelet/pods/d8235a15-af1b-4332-ac83-1aef5bfb1286/volumes" Oct 30 23:58:36.612198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:36.612180 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" path="/var/lib/kubelet/pods/f179a0a0-c464-4a8b-b6cc-04a21610a832/volumes" Oct 30 23:58:36.612767 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:36.612749 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68436c1-d900-4063-b853-cb9be78ceae6" path="/var/lib/kubelet/pods/f68436c1-d900-4063-b853-cb9be78ceae6/volumes" Oct 30 23:58:36.613200 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:36.613183 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb14f0ca-115b-4fe6-afcd-6338df872573" path="/var/lib/kubelet/pods/fb14f0ca-115b-4fe6-afcd-6338df872573/volumes" Oct 30 23:58:38.297850 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:38.297027 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 30 23:58:38.297850 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:38.297788 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:58:38.300260 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:38.300152 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:58:38.300595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:38.300420 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:58:38.301086 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:38.301062 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 30 23:58:38.301243 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:38.301222 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:58:38.301577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:38.301560 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 30 23:58:38.301642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:38.301613 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:58:42.223963 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.223926 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx"] Oct 30 23:58:42.224914 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.224894 7570 cpu_manager.go:414] "containerMap: removing container" podUID="5244915c-9573-489c-b1f8-e22948899df3" containerName="payload-processing" Oct 30 23:58:42.224914 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.224916 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5244915c-9573-489c-b1f8-e22948899df3" containerName="payload-processing" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.224936 7570 cpu_manager.go:414] "containerMap: removing container" podUID="b12dc177-11a1-40b0-9fdf-75993ab176e6" containerName="payload-processing" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.224942 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12dc177-11a1-40b0-9fdf-75993ab176e6" containerName="payload-processing" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.224967 7570 cpu_manager.go:414] "containerMap: removing container" podUID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" containerName="payload-pre-processing" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.224975 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" containerName="payload-pre-processing" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.224985 7570 cpu_manager.go:414] "containerMap: removing container" podUID="12efb180-e881-40a3-9001-8a39a2ddd352" containerName="payload-pre-processing" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.224992 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12efb180-e881-40a3-9001-8a39a2ddd352" containerName="payload-pre-processing" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225012 7570 cpu_manager.go:414] "containerMap: removing container" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerName="istio-proxy" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225018 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerName="istio-proxy" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225032 7570 cpu_manager.go:414] "containerMap: removing container" podUID="d8235a15-af1b-4332-ac83-1aef5bfb1286" containerName="istio-proxy" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225039 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8235a15-af1b-4332-ac83-1aef5bfb1286" containerName="istio-proxy" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225062 7570 cpu_manager.go:414] "containerMap: removing container" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225070 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225086 7570 cpu_manager.go:414] "containerMap: removing container" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" Oct 30 23:58:42.225088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225092 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225106 7570 cpu_manager.go:414] "containerMap: removing container" podUID="f68436c1-d900-4063-b853-cb9be78ceae6" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225114 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68436c1-d900-4063-b853-cb9be78ceae6" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225130 7570 cpu_manager.go:414] "containerMap: removing container" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225135 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225158 7570 cpu_manager.go:414] "containerMap: removing container" podUID="fb14f0ca-115b-4fe6-afcd-6338df872573" containerName="payload-processing" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225166 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb14f0ca-115b-4fe6-afcd-6338df872573" containerName="payload-processing" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225177 7570 cpu_manager.go:414] "containerMap: removing container" podUID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" containerName="payload-processing" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225183 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" containerName="payload-processing" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225204 7570 cpu_manager.go:414] "containerMap: removing container" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225211 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225225 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225231 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225245 7570 cpu_manager.go:414] "containerMap: removing container" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225251 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225260 7570 cpu_manager.go:414] "containerMap: removing container" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225264 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225271 7570 cpu_manager.go:414] "containerMap: removing container" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225275 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225282 7570 cpu_manager.go:414] "containerMap: removing container" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225286 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225293 7570 cpu_manager.go:414] "containerMap: removing container" podUID="a74a48f3-2b68-424b-9489-33e22dab2f2f" containerName="payload-pre-processing" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225298 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74a48f3-2b68-424b-9489-33e22dab2f2f" containerName="payload-pre-processing" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225312 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225316 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225325 7570 cpu_manager.go:414] "containerMap: removing container" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225329 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225479 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="5244915c-9573-489c-b1f8-e22948899df3" containerName="payload-processing" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225491 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12dc177-11a1-40b0-9fdf-75993ab176e6" containerName="payload-processing" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225498 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e6db40-bf7c-4440-bed0-8ec6e70cab0f" containerName="payload-pre-processing" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225507 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="12efb180-e881-40a3-9001-8a39a2ddd352" containerName="payload-pre-processing" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225516 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7f3b9a-978c-49f2-a52a-5293dfb1924f" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225522 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8235a15-af1b-4332-ac83-1aef5bfb1286" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225535 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225544 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225556 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68436c1-d900-4063-b853-cb9be78ceae6" containerName="istio-proxy" Oct 30 23:58:42.226104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225566 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="af49f007-5f21-43cc-b4f3-d78e31b45100" containerName="istio-proxy" Oct 30 23:58:42.227575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225575 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb14f0ca-115b-4fe6-afcd-6338df872573" containerName="payload-processing" Oct 30 23:58:42.227575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225582 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa08b804-5246-4fb8-a68c-cf7af9ba4a54" containerName="payload-processing" Oct 30 23:58:42.227575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225591 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f473a6-9517-4367-b935-294dbd28ceec" containerName="istio-proxy" Oct 30 23:58:42.227575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225599 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ec9765-6ba5-4e0a-ba9f-3ae35dbb7984" containerName="istio-proxy" Oct 30 23:58:42.227575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225609 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e446a0-c6da-40ab-aa1f-f3756e39619a" containerName="istio-proxy" Oct 30 23:58:42.227575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225617 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1017ca7a-a16e-44bd-af55-d8babaa94e01" containerName="istio-proxy" Oct 30 23:58:42.227575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225628 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f179a0a0-c464-4a8b-b6cc-04a21610a832" containerName="istio-proxy" Oct 30 23:58:42.227575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225637 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fc6c4a-5ced-4748-9d6f-c7cede8c7dc7" containerName="istio-proxy" Oct 30 23:58:42.227575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225645 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74a48f3-2b68-424b-9489-33e22dab2f2f" containerName="payload-pre-processing" Oct 30 23:58:42.227575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225655 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c9e5ed-908c-463d-8f4f-395a80ababdd" containerName="payload-pre-processing" Oct 30 23:58:42.227575 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.225665 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94094c6-876a-4082-ad7d-f35bb1909f87" containerName="istio-proxy" Oct 30 23:58:42.230014 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.229996 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:42.237031 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.237007 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:58:42.237794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.237772 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:58:42.289680 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.289656 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx"] Oct 30 23:58:42.340796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.340769 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx\" (UID: \"77945cb3-af14-402a-9c42-faadfcdb9fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:42.340923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.340812 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmbzp\" (UniqueName: \"kubernetes.io/projected/77945cb3-af14-402a-9c42-faadfcdb9fc6-kube-api-access-wmbzp\") pod \"maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx\" (UID: \"77945cb3-af14-402a-9c42-faadfcdb9fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:42.341073 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.341049 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx\" (UID: \"77945cb3-af14-402a-9c42-faadfcdb9fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:42.442771 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.442744 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx\" (UID: \"77945cb3-af14-402a-9c42-faadfcdb9fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:42.442975 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.442958 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx\" (UID: \"77945cb3-af14-402a-9c42-faadfcdb9fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:42.443042 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.442995 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmbzp\" (UniqueName: \"kubernetes.io/projected/77945cb3-af14-402a-9c42-faadfcdb9fc6-kube-api-access-wmbzp\") pod \"maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx\" (UID: \"77945cb3-af14-402a-9c42-faadfcdb9fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:42.443406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.443388 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx\" (UID: \"77945cb3-af14-402a-9c42-faadfcdb9fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:42.449759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.449737 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx\" (UID: \"77945cb3-af14-402a-9c42-faadfcdb9fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:42.455538 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.455518 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmbzp\" (UniqueName: \"kubernetes.io/projected/77945cb3-af14-402a-9c42-faadfcdb9fc6-kube-api-access-wmbzp\") pod \"maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx\" (UID: \"77945cb3-af14-402a-9c42-faadfcdb9fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:42.545611 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.545524 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:42.879612 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:42.879585 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx"] Oct 30 23:58:42.880872 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:42.880844 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77945cb3_af14_402a_9c42_faadfcdb9fc6.slice/crio-ad867f79a6a473bf43867fdeddc1d664070cbb5cf26dde26b72fc105a6d9b6f7 WatchSource:0}: Error finding container ad867f79a6a473bf43867fdeddc1d664070cbb5cf26dde26b72fc105a6d9b6f7: Status 404 returned error can't find the container with id ad867f79a6a473bf43867fdeddc1d664070cbb5cf26dde26b72fc105a6d9b6f7 Oct 30 23:58:43.048539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.048516 7570 scope.go:122] "RemoveContainer" containerID="384a06f41c709dbd8e0b9f506ff9f630aa2e0aa1e4523c8ed6cabb86121c1c89" Oct 30 23:58:43.061315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.061294 7570 scope.go:122] "RemoveContainer" containerID="734e0129e0d5009c2d894c51431c9a5313ff3f54071b2dc5309f546eade589d3" Oct 30 23:58:43.072387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.072371 7570 scope.go:122] "RemoveContainer" containerID="2da79595c41ddcf813b52babb72238b7840eb347770a0979f860ea3c9781803d" Oct 30 23:58:43.085099 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.085080 7570 scope.go:122] "RemoveContainer" containerID="3f64b5a78b92f2c34f681907d33dd8081813506ac9e5ee82e9f44f06a2e6b2fd" Oct 30 23:58:43.105020 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.105001 7570 scope.go:122] "RemoveContainer" containerID="b3c07a3b101c4549f4210027e795ad239cf58b95d2ed5169b8e57030fb403144" Oct 30 23:58:43.116815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.116785 7570 scope.go:122] "RemoveContainer" containerID="4c9d727f8c05504972593ae1b6240121490aba70db54418fa228de5c0ea97248" Oct 30 23:58:43.128058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.128036 7570 scope.go:122] "RemoveContainer" containerID="851e4c679073d2c695b1c06fe3c2af3ff196bc23efffd3ae3aa7db70b8c17ad8" Oct 30 23:58:43.318858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.317354 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-5bhpc" Oct 30 23:58:43.318858 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.318576 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-4blbf" Oct 30 23:58:43.338252 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.338211 7570 kubelet.go:1658] "Container garbage collection succeeded" Oct 30 23:58:43.521316 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.521275 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" event={"ID":"77945cb3-af14-402a-9c42-faadfcdb9fc6","Type":"ContainerStarted","Data":"c227a0741ae0910076f4793372f145e326dc30a40e2f82fb6577a0ab4f499609"} Oct 30 23:58:43.521539 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.521330 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" event={"ID":"77945cb3-af14-402a-9c42-faadfcdb9fc6","Type":"ContainerStarted","Data":"ad867f79a6a473bf43867fdeddc1d664070cbb5cf26dde26b72fc105a6d9b6f7"} Oct 30 23:58:43.563432 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:43.563368 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" podStartSLOduration=1.5633487609999999 podStartE2EDuration="1.563348761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:43.556712047 +0000 UTC m=+2335.075713137" watchObservedRunningTime="2026-10-30 23:58:43.563348761 +0000 UTC m=+2335.082349852" Oct 30 23:58:44.530380 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:44.530348 7570 generic.go:356] "Generic (PLEG): container finished" podID="77945cb3-af14-402a-9c42-faadfcdb9fc6" containerID="c227a0741ae0910076f4793372f145e326dc30a40e2f82fb6577a0ab4f499609" exitCode=0 Oct 30 23:58:44.530777 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:44.530415 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" event={"ID":"77945cb3-af14-402a-9c42-faadfcdb9fc6","Type":"ContainerDied","Data":"c227a0741ae0910076f4793372f145e326dc30a40e2f82fb6577a0ab4f499609"} Oct 30 23:58:45.731901 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:45.731868 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:45.916008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:45.915977 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-xks-ca\") pod \"77945cb3-af14-402a-9c42-faadfcdb9fc6\" (UID: \"77945cb3-af14-402a-9c42-faadfcdb9fc6\") " Oct 30 23:58:45.916174 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:45.916056 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-openshift-ca\") pod \"77945cb3-af14-402a-9c42-faadfcdb9fc6\" (UID: \"77945cb3-af14-402a-9c42-faadfcdb9fc6\") " Oct 30 23:58:45.916254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:45.916241 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77945cb3-af14-402a-9c42-faadfcdb9fc6-kube-api-access-wmbzp\" (UniqueName: \"kubernetes.io/projected/77945cb3-af14-402a-9c42-faadfcdb9fc6-kube-api-access-wmbzp\") pod \"77945cb3-af14-402a-9c42-faadfcdb9fc6\" (UID: \"77945cb3-af14-402a-9c42-faadfcdb9fc6\") " Oct 30 23:58:45.916479 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:45.916432 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-openshift-ca" pod "77945cb3-af14-402a-9c42-faadfcdb9fc6" (UID: "77945cb3-af14-402a-9c42-faadfcdb9fc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:45.916849 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:45.916833 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:45.919974 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:45.919950 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77945cb3-af14-402a-9c42-faadfcdb9fc6-kube-api-access-wmbzp" pod "77945cb3-af14-402a-9c42-faadfcdb9fc6" (UID: "77945cb3-af14-402a-9c42-faadfcdb9fc6"). InnerVolumeSpecName "kube-api-access-wmbzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:45.920380 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:45.920357 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-xks-ca" pod "77945cb3-af14-402a-9c42-faadfcdb9fc6" (UID: "77945cb3-af14-402a-9c42-faadfcdb9fc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:58:46.018919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:46.018884 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmbzp\" (UniqueName: \"kubernetes.io/projected/77945cb3-af14-402a-9c42-faadfcdb9fc6-kube-api-access-wmbzp\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:46.018919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:46.018916 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77945cb3-af14-402a-9c42-faadfcdb9fc6-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:46.585494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:46.583318 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" event={"ID":"77945cb3-af14-402a-9c42-faadfcdb9fc6","Type":"ContainerDied","Data":"ad867f79a6a473bf43867fdeddc1d664070cbb5cf26dde26b72fc105a6d9b6f7"} Oct 30 23:58:46.585494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:46.583393 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad867f79a6a473bf43867fdeddc1d664070cbb5cf26dde26b72fc105a6d9b6f7" Oct 30 23:58:46.585494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:46.583489 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx" Oct 30 23:58:52.168679 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.168648 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm"] Oct 30 23:58:52.169579 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.169555 7570 cpu_manager.go:414] "containerMap: removing container" podUID="77945cb3-af14-402a-9c42-faadfcdb9fc6" containerName="revoke-keys" Oct 30 23:58:52.169579 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.169580 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77945cb3-af14-402a-9c42-faadfcdb9fc6" containerName="revoke-keys" Oct 30 23:58:52.169808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.169794 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="77945cb3-af14-402a-9c42-faadfcdb9fc6" containerName="revoke-keys" Oct 30 23:58:52.173668 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.173653 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:52.181594 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.181569 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:58:52.181726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.181594 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:58:52.183482 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.183445 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm"] Oct 30 23:58:52.270020 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.269987 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9qk\" (UniqueName: \"kubernetes.io/projected/dec6f9e3-832c-422c-8172-123b52f2e2e5-kube-api-access-dm9qk\") pod \"maas-api-revoke-sub-91354e50-6tdnm\" (UID: \"dec6f9e3-832c-422c-8172-123b52f2e2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:52.270233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.270216 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-6tdnm\" (UID: \"dec6f9e3-832c-422c-8172-123b52f2e2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:52.270267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.270248 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-6tdnm\" (UID: \"dec6f9e3-832c-422c-8172-123b52f2e2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:52.372168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.372128 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9qk\" (UniqueName: \"kubernetes.io/projected/dec6f9e3-832c-422c-8172-123b52f2e2e5-kube-api-access-dm9qk\") pod \"maas-api-revoke-sub-91354e50-6tdnm\" (UID: \"dec6f9e3-832c-422c-8172-123b52f2e2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:52.372347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.372330 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-6tdnm\" (UID: \"dec6f9e3-832c-422c-8172-123b52f2e2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:52.372440 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.372367 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-6tdnm\" (UID: \"dec6f9e3-832c-422c-8172-123b52f2e2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:52.373128 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.373105 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-6tdnm\" (UID: \"dec6f9e3-832c-422c-8172-123b52f2e2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:52.374893 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.374869 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-6tdnm\" (UID: \"dec6f9e3-832c-422c-8172-123b52f2e2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:52.401952 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.401929 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9qk\" (UniqueName: \"kubernetes.io/projected/dec6f9e3-832c-422c-8172-123b52f2e2e5-kube-api-access-dm9qk\") pod \"maas-api-revoke-sub-91354e50-6tdnm\" (UID: \"dec6f9e3-832c-422c-8172-123b52f2e2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:52.418761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.418702 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx"] Oct 30 23:58:52.453404 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.453365 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt"] Oct 30 23:58:52.498050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.497856 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:52.823015 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:52.822987 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm"] Oct 30 23:58:52.824742 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:58:52.824695 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec6f9e3_832c_422c_8172_123b52f2e2e5.slice/crio-005c5b4a9cbe86dfc3f27d249f0dcf98e3bfb5e95fead095c61228f81be861b4 WatchSource:0}: Error finding container 005c5b4a9cbe86dfc3f27d249f0dcf98e3bfb5e95fead095c61228f81be861b4: Status 404 returned error can't find the container with id 005c5b4a9cbe86dfc3f27d249f0dcf98e3bfb5e95fead095c61228f81be861b4 Oct 30 23:58:53.674757 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:53.674680 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" event={"ID":"dec6f9e3-832c-422c-8172-123b52f2e2e5","Type":"ContainerStarted","Data":"2cbcced42f15a0d21b9f955bccb9218fbf20b65e04929c9390f0728f19804dd0"} Oct 30 23:58:53.674757 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:53.674731 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" event={"ID":"dec6f9e3-832c-422c-8172-123b52f2e2e5","Type":"ContainerStarted","Data":"005c5b4a9cbe86dfc3f27d249f0dcf98e3bfb5e95fead095c61228f81be861b4"} Oct 30 23:58:53.708878 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:53.708831 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" podStartSLOduration=1.708817397 podStartE2EDuration="1.708817397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:53.704820776 +0000 UTC m=+2345.223821865" watchObservedRunningTime="2026-10-30 23:58:53.708817397 +0000 UTC m=+2345.227818490" Oct 30 23:58:54.683438 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:54.683401 7570 generic.go:356] "Generic (PLEG): container finished" podID="dec6f9e3-832c-422c-8172-123b52f2e2e5" containerID="2cbcced42f15a0d21b9f955bccb9218fbf20b65e04929c9390f0728f19804dd0" exitCode=0 Oct 30 23:58:54.683844 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:54.683448 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" event={"ID":"dec6f9e3-832c-422c-8172-123b52f2e2e5","Type":"ContainerDied","Data":"2cbcced42f15a0d21b9f955bccb9218fbf20b65e04929c9390f0728f19804dd0"} Oct 30 23:58:55.980747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:55.980718 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:56.079082 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:56.079059 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-xks-ca\") pod \"dec6f9e3-832c-422c-8172-123b52f2e2e5\" (UID: \"dec6f9e3-832c-422c-8172-123b52f2e2e5\") " Oct 30 23:58:56.079226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:56.079164 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-openshift-ca\") pod \"dec6f9e3-832c-422c-8172-123b52f2e2e5\" (UID: \"dec6f9e3-832c-422c-8172-123b52f2e2e5\") " Oct 30 23:58:56.079429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:56.079405 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec6f9e3-832c-422c-8172-123b52f2e2e5-kube-api-access-dm9qk\" (UniqueName: \"kubernetes.io/projected/dec6f9e3-832c-422c-8172-123b52f2e2e5-kube-api-access-dm9qk\") pod \"dec6f9e3-832c-422c-8172-123b52f2e2e5\" (UID: \"dec6f9e3-832c-422c-8172-123b52f2e2e5\") " Oct 30 23:58:56.079627 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:56.079603 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-openshift-ca" pod "dec6f9e3-832c-422c-8172-123b52f2e2e5" (UID: "dec6f9e3-832c-422c-8172-123b52f2e2e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:56.080093 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:56.080071 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:56.083700 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:56.083677 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-xks-ca" pod "dec6f9e3-832c-422c-8172-123b52f2e2e5" (UID: "dec6f9e3-832c-422c-8172-123b52f2e2e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:58:56.083786 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:56.083676 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec6f9e3-832c-422c-8172-123b52f2e2e5-kube-api-access-dm9qk" pod "dec6f9e3-832c-422c-8172-123b52f2e2e5" (UID: "dec6f9e3-832c-422c-8172-123b52f2e2e5"). InnerVolumeSpecName "kube-api-access-dm9qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:56.181526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:56.181501 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec6f9e3-832c-422c-8172-123b52f2e2e5-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:56.181526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:56.181524 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm9qk\" (UniqueName: \"kubernetes.io/projected/dec6f9e3-832c-422c-8172-123b52f2e2e5-kube-api-access-dm9qk\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:56.740965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:56.740938 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" Oct 30 23:58:56.741083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:56.740970 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm" event={"ID":"dec6f9e3-832c-422c-8172-123b52f2e2e5","Type":"ContainerDied","Data":"005c5b4a9cbe86dfc3f27d249f0dcf98e3bfb5e95fead095c61228f81be861b4"} Oct 30 23:58:56.741083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:56.741014 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="005c5b4a9cbe86dfc3f27d249f0dcf98e3bfb5e95fead095c61228f81be861b4" Oct 30 23:58:57.419144 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.419109 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" podUID="08022df4-0b83-43af-9b14-b85ab9f83be3" containerName="payload-processing" containerID="cri-o://5586221ccce00e8910c448c68c374e36fb26c7ec0d90934b6f10d1f3ed13ceed" gracePeriod=25 Oct 30 23:58:57.455074 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.455034 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" podUID="1d0b4116-23a4-4074-8dfb-93b7bbf53681" containerName="payload-pre-processing" containerID="cri-o://6d0a6bd4c7aa80c7f055c16f2c9eec203d6315d8f85d57501dd965260c5b3285" gracePeriod=25 Oct 30 23:58:57.756833 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.756795 7570 generic.go:356] "Generic (PLEG): container finished" podID="1d0b4116-23a4-4074-8dfb-93b7bbf53681" containerID="6d0a6bd4c7aa80c7f055c16f2c9eec203d6315d8f85d57501dd965260c5b3285" exitCode=0 Oct 30 23:58:57.756979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.756869 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" event={"ID":"1d0b4116-23a4-4074-8dfb-93b7bbf53681","Type":"ContainerDied","Data":"6d0a6bd4c7aa80c7f055c16f2c9eec203d6315d8f85d57501dd965260c5b3285"} Oct 30 23:58:57.758807 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.758785 7570 generic.go:356] "Generic (PLEG): container finished" podID="08022df4-0b83-43af-9b14-b85ab9f83be3" containerID="5586221ccce00e8910c448c68c374e36fb26c7ec0d90934b6f10d1f3ed13ceed" exitCode=0 Oct 30 23:58:57.758807 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.758809 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" event={"ID":"08022df4-0b83-43af-9b14-b85ab9f83be3","Type":"ContainerDied","Data":"5586221ccce00e8910c448c68c374e36fb26c7ec0d90934b6f10d1f3ed13ceed"} Oct 30 23:58:57.820252 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.820221 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:58:57.846096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.846077 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:58:57.918889 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.918850 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08022df4-0b83-43af-9b14-b85ab9f83be3-kube-api-access-2jmmd\" (UniqueName: \"kubernetes.io/projected/08022df4-0b83-43af-9b14-b85ab9f83be3-kube-api-access-2jmmd\") pod \"08022df4-0b83-43af-9b14-b85ab9f83be3\" (UID: \"08022df4-0b83-43af-9b14-b85ab9f83be3\") " Oct 30 23:58:57.919035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.918897 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08022df4-0b83-43af-9b14-b85ab9f83be3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08022df4-0b83-43af-9b14-b85ab9f83be3-plugins-config-volume\") pod \"08022df4-0b83-43af-9b14-b85ab9f83be3\" (UID: \"08022df4-0b83-43af-9b14-b85ab9f83be3\") " Oct 30 23:58:57.919677 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.919646 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08022df4-0b83-43af-9b14-b85ab9f83be3-plugins-config-volume" pod "08022df4-0b83-43af-9b14-b85ab9f83be3" (UID: "08022df4-0b83-43af-9b14-b85ab9f83be3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:57.920130 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.920106 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08022df4-0b83-43af-9b14-b85ab9f83be3-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:57.921227 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:57.921205 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08022df4-0b83-43af-9b14-b85ab9f83be3-kube-api-access-2jmmd" pod "08022df4-0b83-43af-9b14-b85ab9f83be3" (UID: "08022df4-0b83-43af-9b14-b85ab9f83be3"). InnerVolumeSpecName "kube-api-access-2jmmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:58.021802 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.021733 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0b4116-23a4-4074-8dfb-93b7bbf53681-kube-api-access-n8fwp\" (UniqueName: \"kubernetes.io/projected/1d0b4116-23a4-4074-8dfb-93b7bbf53681-kube-api-access-n8fwp\") pod \"1d0b4116-23a4-4074-8dfb-93b7bbf53681\" (UID: \"1d0b4116-23a4-4074-8dfb-93b7bbf53681\") " Oct 30 23:58:58.021916 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.021827 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0b4116-23a4-4074-8dfb-93b7bbf53681-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0b4116-23a4-4074-8dfb-93b7bbf53681-plugins-config-volume\") pod \"1d0b4116-23a4-4074-8dfb-93b7bbf53681\" (UID: \"1d0b4116-23a4-4074-8dfb-93b7bbf53681\") " Oct 30 23:58:58.022375 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.022329 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0b4116-23a4-4074-8dfb-93b7bbf53681-plugins-config-volume" pod "1d0b4116-23a4-4074-8dfb-93b7bbf53681" (UID: "1d0b4116-23a4-4074-8dfb-93b7bbf53681"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:58.022495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.022427 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jmmd\" (UniqueName: \"kubernetes.io/projected/08022df4-0b83-43af-9b14-b85ab9f83be3-kube-api-access-2jmmd\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:58.024011 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.023990 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0b4116-23a4-4074-8dfb-93b7bbf53681-kube-api-access-n8fwp" pod "1d0b4116-23a4-4074-8dfb-93b7bbf53681" (UID: "1d0b4116-23a4-4074-8dfb-93b7bbf53681"). InnerVolumeSpecName "kube-api-access-n8fwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:58.123931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.123902 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8fwp\" (UniqueName: \"kubernetes.io/projected/1d0b4116-23a4-4074-8dfb-93b7bbf53681-kube-api-access-n8fwp\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:58.123931 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.123934 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0b4116-23a4-4074-8dfb-93b7bbf53681-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:58.743544 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:58.743450 7570 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0b4116_23a4_4074_8dfb_93b7bbf53681.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08022df4_0b83_43af_9b14_b85ab9f83be3.slice/crio-d0d37036f57057b511292cea8c380493d877015b8b3d6c6952b4d755c8f0871c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08022df4_0b83_43af_9b14_b85ab9f83be3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0b4116_23a4_4074_8dfb_93b7bbf53681.slice/crio-c4f694c6311c81e7e237465327a6f8e87cfbce8b7e45e59e5e3e2cf097fc39bb\": RecentStats: unable to find data in memory cache]" Oct 30 23:58:58.767653 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.767625 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" event={"ID":"08022df4-0b83-43af-9b14-b85ab9f83be3","Type":"ContainerDied","Data":"d0d37036f57057b511292cea8c380493d877015b8b3d6c6952b4d755c8f0871c"} Oct 30 23:58:58.767653 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.767639 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx" Oct 30 23:58:58.767840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.767724 7570 scope.go:122] "RemoveContainer" containerID="5586221ccce00e8910c448c68c374e36fb26c7ec0d90934b6f10d1f3ed13ceed" Oct 30 23:58:58.770944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.770918 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" event={"ID":"1d0b4116-23a4-4074-8dfb-93b7bbf53681","Type":"ContainerDied","Data":"c4f694c6311c81e7e237465327a6f8e87cfbce8b7e45e59e5e3e2cf097fc39bb"} Oct 30 23:58:58.771115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.770993 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt" Oct 30 23:58:58.780984 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.780960 7570 scope.go:122] "RemoveContainer" containerID="6d0a6bd4c7aa80c7f055c16f2c9eec203d6315d8f85d57501dd965260c5b3285" Oct 30 23:58:58.790223 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.790199 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx"] Oct 30 23:58:58.794919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.794900 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7fc6e9-5b45467fdf-f6zdx"] Oct 30 23:58:58.808294 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.808273 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt"] Oct 30 23:58:58.813547 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:58:58.813531 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7fc6e9-74bd867bdc-wwstt"] Oct 30 23:58:59.448203 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:58:59.448166 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:59:00.609836 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:00.609804 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08022df4-0b83-43af-9b14-b85ab9f83be3" path="/var/lib/kubelet/pods/08022df4-0b83-43af-9b14-b85ab9f83be3/volumes" Oct 30 23:59:00.610525 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:00.610499 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0b4116-23a4-4074-8dfb-93b7bbf53681" path="/var/lib/kubelet/pods/1d0b4116-23a4-4074-8dfb-93b7bbf53681/volumes" Oct 30 23:59:08.815771 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:08.815729 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx"] Oct 30 23:59:08.817916 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:08.817885 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" containerID="cri-o://12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61" gracePeriod=30 Oct 30 23:59:09.350549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.341054 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx"] Oct 30 23:59:09.350549 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.346085 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fc6e9-nndhx"] Oct 30 23:59:09.433186 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.433141 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:59:09.603851 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.603766 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-metrics-certs\") pod \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " Oct 30 23:59:09.604015 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.603878 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-kube-api-access-jgzmg\" (UniqueName: \"kubernetes.io/projected/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-kube-api-access-jgzmg\") pod \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " Oct 30 23:59:09.604083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.604068 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-openshift-service-ca\") pod \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " Oct 30 23:59:09.604133 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.604118 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls\") pod \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\" (UID: \"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903\") " Oct 30 23:59:09.604429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.604393 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-openshift-service-ca" pod "cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" (UID: "cc2a6d9c-ec3e-438a-9d6b-db75fa30a903"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:09.604816 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.604793 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:09.616134 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.616100 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls" pod "cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" (UID: "cc2a6d9c-ec3e-438a-9d6b-db75fa30a903"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:09.616224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.616134 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-kube-api-access-jgzmg" pod "cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" (UID: "cc2a6d9c-ec3e-438a-9d6b-db75fa30a903"). InnerVolumeSpecName "kube-api-access-jgzmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:09.616224 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.616190 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-metrics-certs" pod "cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" (UID: "cc2a6d9c-ec3e-438a-9d6b-db75fa30a903"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:09.707146 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.707118 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:09.707146 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.707147 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:09.707371 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.707161 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgzmg\" (UniqueName: \"kubernetes.io/projected/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903-kube-api-access-jgzmg\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:09.764588 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.764562 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp"] Oct 30 23:59:09.788955 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.788921 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx"] Oct 30 23:59:09.899258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.899175 7570 generic.go:356] "Generic (PLEG): container finished" podID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerID="12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61" exitCode=0 Oct 30 23:59:09.899804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.899342 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" event={"ID":"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903","Type":"ContainerDied","Data":"12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61"} Oct 30 23:59:09.899804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.899420 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" event={"ID":"cc2a6d9c-ec3e-438a-9d6b-db75fa30a903","Type":"ContainerDied","Data":"f3eb1ef19d45548ba59c05974de3f515521a241a1d6c23b0c274cea205ac46df"} Oct 30 23:59:09.899804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.899449 7570 scope.go:122] "RemoveContainer" containerID="12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61" Oct 30 23:59:09.899804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.899615 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx" Oct 30 23:59:09.917136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.917109 7570 scope.go:122] "RemoveContainer" containerID="d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9" Oct 30 23:59:09.931652 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.931615 7570 scope.go:122] "RemoveContainer" containerID="12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61" Oct 30 23:59:09.932260 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:09.932237 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61\": container with ID starting with 12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61 not found: ID does not exist" containerID="12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61" Oct 30 23:59:09.932359 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.932268 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61"} err="failed to get container status \"12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61\": rpc error: code = NotFound desc = could not find container \"12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61\": container with ID starting with 12873b1d522b1899473c382af45553630e401e9fcb4fb6d799fa7d56be538c61 not found: ID does not exist" Oct 30 23:59:09.932359 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.932292 7570 scope.go:122] "RemoveContainer" containerID="d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9" Oct 30 23:59:09.933599 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:09.932670 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9\": container with ID starting with d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9 not found: ID does not exist" containerID="d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9" Oct 30 23:59:09.933599 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.932717 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9"} err="failed to get container status \"d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9\": rpc error: code = NotFound desc = could not find container \"d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9\": container with ID starting with d0377691fc06ffaf3b2133518d48bbee4fe583259e2120de24af1bc5f9ace5c9 not found: ID does not exist" Oct 30 23:59:09.937804 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.937782 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx"] Oct 30 23:59:09.952338 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:09.952318 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7fc6e9-db4b76bc4-hm7vx"] Oct 30 23:59:10.605887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:10.605847 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77945cb3-af14-402a-9c42-faadfcdb9fc6" path="/var/lib/kubelet/pods/77945cb3-af14-402a-9c42-faadfcdb9fc6/volumes" Oct 30 23:59:10.606505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:10.606480 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" path="/var/lib/kubelet/pods/cc2a6d9c-ec3e-438a-9d6b-db75fa30a903/volumes" Oct 30 23:59:10.949400 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:10.949322 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl"] Oct 30 23:59:13.439376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:13.438578 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp"] Oct 30 23:59:13.439376 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:13.438789 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" containerID="cri-o://a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588" gracePeriod=30 Oct 30 23:59:13.445794 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:13.445773 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd"] Oct 30 23:59:13.445961 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:13.445941 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerName="istio-proxy" containerID="cri-o://bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf" gracePeriod=30 Oct 30 23:59:13.452647 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:13.452623 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr"] Oct 30 23:59:13.452836 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:13.452815 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" containerID="cri-o://5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854" gracePeriod=30 Oct 30 23:59:14.768867 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:14.765668 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" podUID="e0f6ac75-7dd2-44be-8b54-92ff018b8a3d" containerName="payload-processing" containerID="cri-o://b790873757bf25b53bae157fa4d35b942dfd574013d0a1f057ee375f4e365537" gracePeriod=25 Oct 30 23:59:14.790166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:14.789868 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" podUID="6843e367-692b-41f0-a9c6-7704d732f009" containerName="payload-pre-processing" containerID="cri-o://14e282ade51dac8dc806a37d0545f5277b669f6986fb50c4a208641e5ab6f498" gracePeriod=25 Oct 30 23:59:14.950296 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:14.950267 7570 generic.go:356] "Generic (PLEG): container finished" podID="6843e367-692b-41f0-a9c6-7704d732f009" containerID="14e282ade51dac8dc806a37d0545f5277b669f6986fb50c4a208641e5ab6f498" exitCode=0 Oct 30 23:59:14.950597 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:14.950575 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" event={"ID":"6843e367-692b-41f0-a9c6-7704d732f009","Type":"ContainerDied","Data":"14e282ade51dac8dc806a37d0545f5277b669f6986fb50c4a208641e5ab6f498"} Oct 30 23:59:14.956839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:14.956814 7570 generic.go:356] "Generic (PLEG): container finished" podID="e0f6ac75-7dd2-44be-8b54-92ff018b8a3d" containerID="b790873757bf25b53bae157fa4d35b942dfd574013d0a1f057ee375f4e365537" exitCode=0 Oct 30 23:59:14.956955 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:14.956855 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" event={"ID":"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d","Type":"ContainerDied","Data":"b790873757bf25b53bae157fa4d35b942dfd574013d0a1f057ee375f4e365537"} Oct 30 23:59:15.162435 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.162409 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:59:15.204789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.204749 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:59:15.290895 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.290867 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6843e367-692b-41f0-a9c6-7704d732f009-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6843e367-692b-41f0-a9c6-7704d732f009-plugins-config-volume\") pod \"6843e367-692b-41f0-a9c6-7704d732f009\" (UID: \"6843e367-692b-41f0-a9c6-7704d732f009\") " Oct 30 23:59:15.291103 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.291022 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-kube-api-access-w4bhr\" (UniqueName: \"kubernetes.io/projected/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-kube-api-access-w4bhr\") pod \"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d\" (UID: \"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d\") " Oct 30 23:59:15.291194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.291135 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6843e367-692b-41f0-a9c6-7704d732f009-kube-api-access-lbmf8\" (UniqueName: \"kubernetes.io/projected/6843e367-692b-41f0-a9c6-7704d732f009-kube-api-access-lbmf8\") pod \"6843e367-692b-41f0-a9c6-7704d732f009\" (UID: \"6843e367-692b-41f0-a9c6-7704d732f009\") " Oct 30 23:59:15.291258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.291201 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6843e367-692b-41f0-a9c6-7704d732f009-plugins-config-volume" pod "6843e367-692b-41f0-a9c6-7704d732f009" (UID: "6843e367-692b-41f0-a9c6-7704d732f009"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:15.291258 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.291236 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-plugins-config-volume\") pod \"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d\" (UID: \"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d\") " Oct 30 23:59:15.291567 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.291544 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-plugins-config-volume" pod "e0f6ac75-7dd2-44be-8b54-92ff018b8a3d" (UID: "e0f6ac75-7dd2-44be-8b54-92ff018b8a3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:15.291953 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.291935 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:15.292044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.291956 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6843e367-692b-41f0-a9c6-7704d732f009-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:15.294184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.294158 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-kube-api-access-w4bhr" pod "e0f6ac75-7dd2-44be-8b54-92ff018b8a3d" (UID: "e0f6ac75-7dd2-44be-8b54-92ff018b8a3d"). InnerVolumeSpecName "kube-api-access-w4bhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:15.294272 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.294231 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6843e367-692b-41f0-a9c6-7704d732f009-kube-api-access-lbmf8" pod "6843e367-692b-41f0-a9c6-7704d732f009" (UID: "6843e367-692b-41f0-a9c6-7704d732f009"). InnerVolumeSpecName "kube-api-access-lbmf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:15.393307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.393279 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4bhr\" (UniqueName: \"kubernetes.io/projected/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d-kube-api-access-w4bhr\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:15.393307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.393308 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbmf8\" (UniqueName: \"kubernetes.io/projected/6843e367-692b-41f0-a9c6-7704d732f009-kube-api-access-lbmf8\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:15.966526 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.966502 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" Oct 30 23:59:15.966890 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.966499 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp" event={"ID":"e0f6ac75-7dd2-44be-8b54-92ff018b8a3d","Type":"ContainerDied","Data":"7558b12fd5d8ff18a6d85e355482072e59d6decd7da49a152728deb9ad9d1150"} Oct 30 23:59:15.966890 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.966660 7570 scope.go:122] "RemoveContainer" containerID="b790873757bf25b53bae157fa4d35b942dfd574013d0a1f057ee375f4e365537" Oct 30 23:59:15.968179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.968159 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" event={"ID":"6843e367-692b-41f0-a9c6-7704d732f009","Type":"ContainerDied","Data":"9ed163a296ee9db20106e8dd23c424d0fcc7fbda9a675c853fd1d7b39783e95b"} Oct 30 23:59:15.968296 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.968204 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx" Oct 30 23:59:15.981714 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:15.981697 7570 scope.go:122] "RemoveContainer" containerID="14e282ade51dac8dc806a37d0545f5277b669f6986fb50c4a208641e5ab6f498" Oct 30 23:59:16.002050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:16.002016 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx"] Oct 30 23:59:16.006678 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:16.006657 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a63288-d588b4f5f-9lzqx"] Oct 30 23:59:16.022544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:16.021362 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp"] Oct 30 23:59:16.025671 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:16.025653 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a63288-79bbd479c-8njpp"] Oct 30 23:59:16.357527 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:16.357493 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp"] Oct 30 23:59:16.606027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:16.606000 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6843e367-692b-41f0-a9c6-7704d732f009" path="/var/lib/kubelet/pods/6843e367-692b-41f0-a9c6-7704d732f009/volumes" Oct 30 23:59:16.606634 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:16.606612 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f6ac75-7dd2-44be-8b54-92ff018b8a3d" path="/var/lib/kubelet/pods/e0f6ac75-7dd2-44be-8b54-92ff018b8a3d/volumes" Oct 30 23:59:18.822793 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.822770 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:59:18.874529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.874215 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istiod-ca-cert\") pod \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " Oct 30 23:59:18.874529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.874292 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-podinfo\") pod \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " Oct 30 23:59:18.874529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.874324 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-envoy\") pod \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " Oct 30 23:59:18.874529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.874435 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-credential-socket\") pod \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " Oct 30 23:59:18.874887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.874573 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-token\" (UniqueName: \"kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-token\") pod \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " Oct 30 23:59:18.874887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.874628 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-certs\") pod \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " Oct 30 23:59:18.874887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.874677 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-kube-api-access-dmfq5\" (UniqueName: \"kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-kube-api-access-dmfq5\") pod \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " Oct 30 23:59:18.874887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.874735 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-data\") pod \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " Oct 30 23:59:18.875078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.874899 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-socket\") pod \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\" (UID: \"cbf9885e-6fe7-4c33-b91a-60b11bf4b730\") " Oct 30 23:59:18.875881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.875841 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-socket" pod "cbf9885e-6fe7-4c33-b91a-60b11bf4b730" (UID: "cbf9885e-6fe7-4c33-b91a-60b11bf4b730"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.876068 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.876048 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-certs" pod "cbf9885e-6fe7-4c33-b91a-60b11bf4b730" (UID: "cbf9885e-6fe7-4c33-b91a-60b11bf4b730"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.878355 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.878322 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-token" pod "cbf9885e-6fe7-4c33-b91a-60b11bf4b730" (UID: "cbf9885e-6fe7-4c33-b91a-60b11bf4b730"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:18.881541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.881443 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-podinfo" pod "cbf9885e-6fe7-4c33-b91a-60b11bf4b730" (UID: "cbf9885e-6fe7-4c33-b91a-60b11bf4b730"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:59:18.882516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.881840 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-credential-socket" pod "cbf9885e-6fe7-4c33-b91a-60b11bf4b730" (UID: "cbf9885e-6fe7-4c33-b91a-60b11bf4b730"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.882516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.882035 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-data" pod "cbf9885e-6fe7-4c33-b91a-60b11bf4b730" (UID: "cbf9885e-6fe7-4c33-b91a-60b11bf4b730"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.890839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.883120 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-envoy" pod "cbf9885e-6fe7-4c33-b91a-60b11bf4b730" (UID: "cbf9885e-6fe7-4c33-b91a-60b11bf4b730"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.890839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.883822 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-kube-api-access-dmfq5" pod "cbf9885e-6fe7-4c33-b91a-60b11bf4b730" (UID: "cbf9885e-6fe7-4c33-b91a-60b11bf4b730"). InnerVolumeSpecName "kube-api-access-dmfq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:18.890839 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.889692 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istiod-ca-cert" pod "cbf9885e-6fe7-4c33-b91a-60b11bf4b730" (UID: "cbf9885e-6fe7-4c33-b91a-60b11bf4b730"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:18.896256 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.895727 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:59:18.937712 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.925741 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:59:18.976286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976252 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb95af9b-7595-4dd9-839a-e698e2a22c07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb95af9b-7595-4dd9-839a-e698e2a22c07-istiod-ca-cert\") pod \"cb95af9b-7595-4dd9-839a-e698e2a22c07\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " Oct 30 23:59:18.976541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976298 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-data\") pod \"cb95af9b-7595-4dd9-839a-e698e2a22c07\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " Oct 30 23:59:18.976541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976340 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-credential-socket\") pod \"cb95af9b-7595-4dd9-839a-e698e2a22c07\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " Oct 30 23:59:18.976541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976394 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-certs\") pod \"cb95af9b-7595-4dd9-839a-e698e2a22c07\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " Oct 30 23:59:18.976541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976510 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-podinfo\") pod \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " Oct 30 23:59:18.976761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976609 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istiod-ca-cert\") pod \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " Oct 30 23:59:18.976761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976635 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-token\" (UniqueName: \"kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-token\") pod \"cb95af9b-7595-4dd9-839a-e698e2a22c07\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " Oct 30 23:59:18.976761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976674 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-data\") pod \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " Oct 30 23:59:18.976761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976736 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-podinfo\") pod \"cb95af9b-7595-4dd9-839a-e698e2a22c07\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " Oct 30 23:59:18.976761 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976759 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-socket\") pod \"cb95af9b-7595-4dd9-839a-e698e2a22c07\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " Oct 30 23:59:18.976985 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976782 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-envoy\") pod \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " Oct 30 23:59:18.976985 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976816 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-certs\") pod \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " Oct 30 23:59:18.976985 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976865 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-kube-api-access-44hlq\" (UniqueName: \"kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-kube-api-access-44hlq\") pod \"cb95af9b-7595-4dd9-839a-e698e2a22c07\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " Oct 30 23:59:18.976985 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976892 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-socket\") pod \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " Oct 30 23:59:18.976985 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976935 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-kube-api-access-xbd7c\" (UniqueName: \"kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-kube-api-access-xbd7c\") pod \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " Oct 30 23:59:18.976985 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976962 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-envoy\") pod \"cb95af9b-7595-4dd9-839a-e698e2a22c07\" (UID: \"cb95af9b-7595-4dd9-839a-e698e2a22c07\") " Oct 30 23:59:18.977192 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.976991 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-token\" (UniqueName: \"kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-token\") pod \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " Oct 30 23:59:18.977192 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.977021 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-credential-socket\") pod \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\" (UID: \"f8b86c0c-3f2f-4f9d-be26-948cb9084373\") " Oct 30 23:59:18.977430 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.977407 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-data" pod "f8b86c0c-3f2f-4f9d-be26-948cb9084373" (UID: "f8b86c0c-3f2f-4f9d-be26-948cb9084373"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.977665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.977644 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-credential-socket" pod "f8b86c0c-3f2f-4f9d-be26-948cb9084373" (UID: "f8b86c0c-3f2f-4f9d-be26-948cb9084373"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.977727 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.977665 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istiod-ca-cert" pod "f8b86c0c-3f2f-4f9d-be26-948cb9084373" (UID: "f8b86c0c-3f2f-4f9d-be26-948cb9084373"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:18.977972 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.977945 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmfq5\" (UniqueName: \"kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-kube-api-access-dmfq5\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.978078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.977976 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.978078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.977990 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.978078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.978003 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.978078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.978016 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.978078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.978029 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.978078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.978042 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.978078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.978056 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.978078 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.978069 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.978540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.978082 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.978540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.978094 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.978540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.978105 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf9885e-6fe7-4c33-b91a-60b11bf4b730-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.978540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.978390 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-socket" pod "f8b86c0c-3f2f-4f9d-be26-948cb9084373" (UID: "f8b86c0c-3f2f-4f9d-be26-948cb9084373"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.978752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.978624 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-certs" pod "f8b86c0c-3f2f-4f9d-be26-948cb9084373" (UID: "f8b86c0c-3f2f-4f9d-be26-948cb9084373"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.979665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.979105 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-certs" pod "cb95af9b-7595-4dd9-839a-e698e2a22c07" (UID: "cb95af9b-7595-4dd9-839a-e698e2a22c07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.979665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.979378 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-data" pod "cb95af9b-7595-4dd9-839a-e698e2a22c07" (UID: "cb95af9b-7595-4dd9-839a-e698e2a22c07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.981122 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.981093 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-podinfo" pod "f8b86c0c-3f2f-4f9d-be26-948cb9084373" (UID: "f8b86c0c-3f2f-4f9d-be26-948cb9084373"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:59:18.981970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.981910 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-credential-socket" pod "cb95af9b-7595-4dd9-839a-e698e2a22c07" (UID: "cb95af9b-7595-4dd9-839a-e698e2a22c07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.981970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.981930 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-socket" pod "cb95af9b-7595-4dd9-839a-e698e2a22c07" (UID: "cb95af9b-7595-4dd9-839a-e698e2a22c07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.982385 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.982362 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb95af9b-7595-4dd9-839a-e698e2a22c07-istiod-ca-cert" pod "cb95af9b-7595-4dd9-839a-e698e2a22c07" (UID: "cb95af9b-7595-4dd9-839a-e698e2a22c07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:18.984252 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.984226 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-podinfo" pod "cb95af9b-7595-4dd9-839a-e698e2a22c07" (UID: "cb95af9b-7595-4dd9-839a-e698e2a22c07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:59:18.991871 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.991825 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-token" pod "cb95af9b-7595-4dd9-839a-e698e2a22c07" (UID: "cb95af9b-7595-4dd9-839a-e698e2a22c07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:18.992455 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.992430 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-token" pod "f8b86c0c-3f2f-4f9d-be26-948cb9084373" (UID: "f8b86c0c-3f2f-4f9d-be26-948cb9084373"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:18.992921 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.992868 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-kube-api-access-44hlq" pod "cb95af9b-7595-4dd9-839a-e698e2a22c07" (UID: "cb95af9b-7595-4dd9-839a-e698e2a22c07"). InnerVolumeSpecName "kube-api-access-44hlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:18.992921 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.992903 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-envoy" pod "f8b86c0c-3f2f-4f9d-be26-948cb9084373" (UID: "f8b86c0c-3f2f-4f9d-be26-948cb9084373"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.993084 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.992967 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-kube-api-access-xbd7c" pod "f8b86c0c-3f2f-4f9d-be26-948cb9084373" (UID: "f8b86c0c-3f2f-4f9d-be26-948cb9084373"). InnerVolumeSpecName "kube-api-access-xbd7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:18.993348 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:18.993323 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-envoy" pod "cb95af9b-7595-4dd9-839a-e698e2a22c07" (UID: "cb95af9b-7595-4dd9-839a-e698e2a22c07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:19.006534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.006511 7570 generic.go:356] "Generic (PLEG): container finished" podID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerID="a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588" exitCode=0 Oct 30 23:59:19.006646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.006571 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" Oct 30 23:59:19.006646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.006598 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" event={"ID":"cb95af9b-7595-4dd9-839a-e698e2a22c07","Type":"ContainerDied","Data":"a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588"} Oct 30 23:59:19.006748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.006659 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp" event={"ID":"cb95af9b-7595-4dd9-839a-e698e2a22c07","Type":"ContainerDied","Data":"7810fe887752c1407d5b0679f1d7482ddd808493153ddc765c93b379508e7ce7"} Oct 30 23:59:19.006748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.006674 7570 scope.go:122] "RemoveContainer" containerID="a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588" Oct 30 23:59:19.008520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.008286 7570 generic.go:356] "Generic (PLEG): container finished" podID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerID="5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854" exitCode=0 Oct 30 23:59:19.008520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.008339 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" Oct 30 23:59:19.008520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.008386 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" event={"ID":"f8b86c0c-3f2f-4f9d-be26-948cb9084373","Type":"ContainerDied","Data":"5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854"} Oct 30 23:59:19.008520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.008504 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr" event={"ID":"f8b86c0c-3f2f-4f9d-be26-948cb9084373","Type":"ContainerDied","Data":"d227395e13ef5d3712a842aa0cba775cedb5e82bed38de62721b43a42c08edeb"} Oct 30 23:59:19.010934 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.010811 7570 generic.go:356] "Generic (PLEG): container finished" podID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerID="bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf" exitCode=0 Oct 30 23:59:19.010934 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.010858 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" Oct 30 23:59:19.010934 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.010869 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" event={"ID":"cbf9885e-6fe7-4c33-b91a-60b11bf4b730","Type":"ContainerDied","Data":"bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf"} Oct 30 23:59:19.010934 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.010904 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd" event={"ID":"cbf9885e-6fe7-4c33-b91a-60b11bf4b730","Type":"ContainerDied","Data":"ba6cb2374b8020117f144a68b9dc8fbe6cfa9f0dd1dc298dc4291f95a13c306e"} Oct 30 23:59:19.029437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.028816 7570 scope.go:122] "RemoveContainer" containerID="a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588" Oct 30 23:59:19.029437 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:19.029108 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588\": container with ID starting with a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588 not found: ID does not exist" containerID="a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588" Oct 30 23:59:19.029437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.029140 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588"} err="failed to get container status \"a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588\": rpc error: code = NotFound desc = could not find container \"a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588\": container with ID starting with a1b9a3ce927dce731cac44b6e11ec5953091246971a34cc7e52a73ec3812e588 not found: ID does not exist" Oct 30 23:59:19.029437 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.029159 7570 scope.go:122] "RemoveContainer" containerID="5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854" Oct 30 23:59:19.033331 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.033306 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt"] Oct 30 23:59:19.044557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.044537 7570 scope.go:122] "RemoveContainer" containerID="5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854" Oct 30 23:59:19.044885 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:19.044865 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854\": container with ID starting with 5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854 not found: ID does not exist" containerID="5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854" Oct 30 23:59:19.044951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.044895 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854"} err="failed to get container status \"5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854\": rpc error: code = NotFound desc = could not find container \"5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854\": container with ID starting with 5aa3d064860981dfe2825bbf293e2688d4443bf8828cf6df9926488835f51854 not found: ID does not exist" Oct 30 23:59:19.044951 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.044918 7570 scope.go:122] "RemoveContainer" containerID="bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf" Oct 30 23:59:19.057000 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.056967 7570 scope.go:122] "RemoveContainer" containerID="bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf" Oct 30 23:59:19.057637 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:19.057614 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf\": container with ID starting with bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf not found: ID does not exist" containerID="bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf" Oct 30 23:59:19.057728 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.057642 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf"} err="failed to get container status \"bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf\": rpc error: code = NotFound desc = could not find container \"bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf\": container with ID starting with bf96cc21067bdc862f72ae443d8ee3f42313bc6ca49469245b27db8874d3b3cf not found: ID does not exist" Oct 30 23:59:19.062179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.062157 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp"] Oct 30 23:59:19.071137 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.071115 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-mh2zp"] Oct 30 23:59:19.079625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079600 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb95af9b-7595-4dd9-839a-e698e2a22c07-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079625 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079624 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079634 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079642 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079651 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079660 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079673 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079688 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079731 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079741 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079750 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44hlq\" (UniqueName: \"kubernetes.io/projected/cb95af9b-7595-4dd9-839a-e698e2a22c07-kube-api-access-44hlq\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079760 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b86c0c-3f2f-4f9d-be26-948cb9084373-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079768 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbd7c\" (UniqueName: \"kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-kube-api-access-xbd7c\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079777 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb95af9b-7595-4dd9-839a-e698e2a22c07-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.079780 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.079785 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b86c0c-3f2f-4f9d-be26-948cb9084373-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.088491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.088455 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd"] Oct 30 23:59:19.098619 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.098599 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qlqnd"] Oct 30 23:59:19.151968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.151936 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr"] Oct 30 23:59:19.157637 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.157609 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-gd9qr"] Oct 30 23:59:19.838890 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.838862 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb"] Oct 30 23:59:19.839295 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:19.839069 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" containerID="cri-o://5cbc0db062b1b3fe35cd3806d520010a27549f62c95551106e51ac669550b876" gracePeriod=30 Oct 30 23:59:20.021564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.021521 7570 generic.go:356] "Generic (PLEG): container finished" podID="4af4419c-fffd-463e-9577-94bad299abc6" containerID="5cbc0db062b1b3fe35cd3806d520010a27549f62c95551106e51ac669550b876" exitCode=0 Oct 30 23:59:20.021722 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.021587 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" event={"ID":"4af4419c-fffd-463e-9577-94bad299abc6","Type":"ContainerDied","Data":"5cbc0db062b1b3fe35cd3806d520010a27549f62c95551106e51ac669550b876"} Oct 30 23:59:20.021722 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.021665 7570 scope.go:122] "RemoveContainer" containerID="7109ce1e7cdee0e506b5064f13410fff49b1fa11c0d3b80c387c43b8850f7c5f" Oct 30 23:59:20.197854 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.197824 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:59:20.384358 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.384269 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af4419c-fffd-463e-9577-94bad299abc6-kube-api-access-52j88\" (UniqueName: \"kubernetes.io/projected/4af4419c-fffd-463e-9577-94bad299abc6-kube-api-access-52j88\") pod \"4af4419c-fffd-463e-9577-94bad299abc6\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " Oct 30 23:59:20.384358 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.384346 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-metrics-certs\") pod \"4af4419c-fffd-463e-9577-94bad299abc6\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " Oct 30 23:59:20.384644 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.384584 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af4419c-fffd-463e-9577-94bad299abc6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4af4419c-fffd-463e-9577-94bad299abc6-openshift-service-ca\") pod \"4af4419c-fffd-463e-9577-94bad299abc6\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " Oct 30 23:59:20.384720 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.384703 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-maas-api-tls\") pod \"4af4419c-fffd-463e-9577-94bad299abc6\" (UID: \"4af4419c-fffd-463e-9577-94bad299abc6\") " Oct 30 23:59:20.384979 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.384956 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af4419c-fffd-463e-9577-94bad299abc6-openshift-service-ca" pod "4af4419c-fffd-463e-9577-94bad299abc6" (UID: "4af4419c-fffd-463e-9577-94bad299abc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:20.385401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.385336 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4af4419c-fffd-463e-9577-94bad299abc6-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:20.388056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.388030 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-maas-api-tls" pod "4af4419c-fffd-463e-9577-94bad299abc6" (UID: "4af4419c-fffd-463e-9577-94bad299abc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:20.399565 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.399540 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-metrics-certs" pod "4af4419c-fffd-463e-9577-94bad299abc6" (UID: "4af4419c-fffd-463e-9577-94bad299abc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:20.399565 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.399549 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af4419c-fffd-463e-9577-94bad299abc6-kube-api-access-52j88" pod "4af4419c-fffd-463e-9577-94bad299abc6" (UID: "4af4419c-fffd-463e-9577-94bad299abc6"). InnerVolumeSpecName "kube-api-access-52j88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:20.487440 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.487405 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:20.487568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.487445 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52j88\" (UniqueName: \"kubernetes.io/projected/4af4419c-fffd-463e-9577-94bad299abc6-kube-api-access-52j88\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:20.487568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.487481 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4af4419c-fffd-463e-9577-94bad299abc6-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:20.604577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.604546 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" path="/var/lib/kubelet/pods/cb95af9b-7595-4dd9-839a-e698e2a22c07/volumes" Oct 30 23:59:20.605125 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.605104 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" path="/var/lib/kubelet/pods/cbf9885e-6fe7-4c33-b91a-60b11bf4b730/volumes" Oct 30 23:59:20.605808 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:20.605786 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" path="/var/lib/kubelet/pods/f8b86c0c-3f2f-4f9d-be26-948cb9084373/volumes" Oct 30 23:59:21.034623 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:21.034589 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" Oct 30 23:59:21.035055 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:21.034631 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb" event={"ID":"4af4419c-fffd-463e-9577-94bad299abc6","Type":"ContainerDied","Data":"148966b0684dd6a7e73039ee4833512c2faa9fbf331d183d128cd76647e653ec"} Oct 30 23:59:21.035055 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:21.034701 7570 scope.go:122] "RemoveContainer" containerID="5cbc0db062b1b3fe35cd3806d520010a27549f62c95551106e51ac669550b876" Oct 30 23:59:21.061641 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:21.061609 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb"] Oct 30 23:59:21.066196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:21.066178 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6c5084-6f79fb6bdf-xmbmb"] Oct 30 23:59:21.358644 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:21.358607 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" podUID="522aa71b-47e4-44b1-825c-bf81305d21b4" containerName="payload-pre-processing" containerID="cri-o://bf345e06964e0f29e4f25c50b80cc58ae8f87f140017f2f23d4fa69b57312ece" gracePeriod=25 Oct 30 23:59:21.752510 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:21.752394 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc"] Oct 30 23:59:21.752722 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:21.752697 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" containerID="cri-o://b68cf4f3a12e3c954ae3cd6c6dc399443a0cfd9ec46ace8c996500adcef15c2c" gracePeriod=30 Oct 30 23:59:22.223026 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.222853 7570 generic.go:356] "Generic (PLEG): container finished" podID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerID="b68cf4f3a12e3c954ae3cd6c6dc399443a0cfd9ec46ace8c996500adcef15c2c" exitCode=0 Oct 30 23:59:22.223026 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.222926 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" event={"ID":"6b3d259c-4f6c-4033-b6dd-1059616fe3b4","Type":"ContainerDied","Data":"b68cf4f3a12e3c954ae3cd6c6dc399443a0cfd9ec46ace8c996500adcef15c2c"} Oct 30 23:59:22.223026 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.222965 7570 scope.go:122] "RemoveContainer" containerID="a122980b7cbc76a173c9f98de07c0e43bb457617d98005f8f366d95243978ad7" Oct 30 23:59:22.224927 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.224913 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:59:22.405973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.405880 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-maas-api-tls\") pod \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " Oct 30 23:59:22.405973 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.405952 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-openshift-service-ca\") pod \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " Oct 30 23:59:22.406206 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.406074 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-kube-api-access-xcbvg\" (UniqueName: \"kubernetes.io/projected/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-kube-api-access-xcbvg\") pod \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " Oct 30 23:59:22.406422 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.406400 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-metrics-certs\") pod \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\" (UID: \"6b3d259c-4f6c-4033-b6dd-1059616fe3b4\") " Oct 30 23:59:22.406525 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.406428 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-openshift-service-ca" pod "6b3d259c-4f6c-4033-b6dd-1059616fe3b4" (UID: "6b3d259c-4f6c-4033-b6dd-1059616fe3b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:22.407194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.407166 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:22.411908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.411864 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-kube-api-access-xcbvg" pod "6b3d259c-4f6c-4033-b6dd-1059616fe3b4" (UID: "6b3d259c-4f6c-4033-b6dd-1059616fe3b4"). InnerVolumeSpecName "kube-api-access-xcbvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:22.412969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.412936 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-metrics-certs" pod "6b3d259c-4f6c-4033-b6dd-1059616fe3b4" (UID: "6b3d259c-4f6c-4033-b6dd-1059616fe3b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:22.414686 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.414643 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-maas-api-tls" pod "6b3d259c-4f6c-4033-b6dd-1059616fe3b4" (UID: "6b3d259c-4f6c-4033-b6dd-1059616fe3b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:22.512860 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.512817 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:22.512860 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.512857 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcbvg\" (UniqueName: \"kubernetes.io/projected/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-kube-api-access-xcbvg\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:22.513066 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.512875 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3d259c-4f6c-4033-b6dd-1059616fe3b4-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:22.605637 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:22.605604 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4af4419c-fffd-463e-9577-94bad299abc6" path="/var/lib/kubelet/pods/4af4419c-fffd-463e-9577-94bad299abc6/volumes" Oct 30 23:59:23.187610 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:23.187574 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" podUID="522aa71b-47e4-44b1-825c-bf81305d21b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 23:59:23.242739 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:23.242711 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" Oct 30 23:59:23.243351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:23.242861 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc" event={"ID":"6b3d259c-4f6c-4033-b6dd-1059616fe3b4","Type":"ContainerDied","Data":"e2166af4fede7416e20d757b1ece44a9283660c9ee1544125db3c86378b99b4f"} Oct 30 23:59:23.243450 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:23.243393 7570 scope.go:122] "RemoveContainer" containerID="b68cf4f3a12e3c954ae3cd6c6dc399443a0cfd9ec46ace8c996500adcef15c2c" Oct 30 23:59:23.292629 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:23.292600 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc"] Oct 30 23:59:23.305840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:23.305807 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a63288-6cbfb8668b-f8spc"] Oct 30 23:59:24.567518 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:24.567482 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt"] Oct 30 23:59:24.605542 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:24.605514 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" path="/var/lib/kubelet/pods/6b3d259c-4f6c-4033-b6dd-1059616fe3b4/volumes" Oct 30 23:59:25.994557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:25.994516 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" containerID="cri-o://4d5e41c0d0ab89419a20ad13530d6b08ee78346448059542a0475a7805ffd355" gracePeriod=45 Oct 30 23:59:26.270756 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.270729 7570 generic.go:356] "Generic (PLEG): container finished" podID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerID="4d5e41c0d0ab89419a20ad13530d6b08ee78346448059542a0475a7805ffd355" exitCode=0 Oct 30 23:59:26.270880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.270802 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" event={"ID":"ee6148fe-e516-492b-b5a7-755a75e82c58","Type":"ContainerDied","Data":"4d5e41c0d0ab89419a20ad13530d6b08ee78346448059542a0475a7805ffd355"} Oct 30 23:59:26.270880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.270851 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" event={"ID":"ee6148fe-e516-492b-b5a7-755a75e82c58","Type":"ContainerDied","Data":"fd5dcc61353c067fac281a25326b737d1e5496fa9cc24805ecf7a72cadb88df2"} Oct 30 23:59:26.270880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.270869 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd5dcc61353c067fac281a25326b737d1e5496fa9cc24805ecf7a72cadb88df2" Oct 30 23:59:26.285108 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.285085 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:59:26.393970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.393944 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6148fe-e516-492b-b5a7-755a75e82c58-kube-api-access-qmfqx\" (UniqueName: \"kubernetes.io/projected/ee6148fe-e516-492b-b5a7-755a75e82c58-kube-api-access-qmfqx\") pod \"ee6148fe-e516-492b-b5a7-755a75e82c58\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " Oct 30 23:59:26.394113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.394016 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-home\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-home\") pod \"ee6148fe-e516-492b-b5a7-755a75e82c58\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " Oct 30 23:59:26.394113 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.394070 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee6148fe-e516-492b-b5a7-755a75e82c58-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6148fe-e516-492b-b5a7-755a75e82c58-tls-certs\") pod \"ee6148fe-e516-492b-b5a7-755a75e82c58\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " Oct 30 23:59:26.394272 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.394249 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-tmp-dir\") pod \"ee6148fe-e516-492b-b5a7-755a75e82c58\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " Oct 30 23:59:26.394405 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.394314 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-home" pod "ee6148fe-e516-492b-b5a7-755a75e82c58" (UID: "ee6148fe-e516-492b-b5a7-755a75e82c58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:26.394405 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.394361 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-dshm\") pod \"ee6148fe-e516-492b-b5a7-755a75e82c58\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " Oct 30 23:59:26.394558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.394391 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-model-cache\") pod \"ee6148fe-e516-492b-b5a7-755a75e82c58\" (UID: \"ee6148fe-e516-492b-b5a7-755a75e82c58\") " Oct 30 23:59:26.394558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.394518 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-tmp-dir" pod "ee6148fe-e516-492b-b5a7-755a75e82c58" (UID: "ee6148fe-e516-492b-b5a7-755a75e82c58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:26.395175 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.395154 7570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-home\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:26.395292 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.395202 7570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-tmp-dir\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:26.395292 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.395157 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-model-cache" pod "ee6148fe-e516-492b-b5a7-755a75e82c58" (UID: "ee6148fe-e516-492b-b5a7-755a75e82c58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:26.396720 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.396697 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee6148fe-e516-492b-b5a7-755a75e82c58-tls-certs" pod "ee6148fe-e516-492b-b5a7-755a75e82c58" (UID: "ee6148fe-e516-492b-b5a7-755a75e82c58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:26.397835 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.397813 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-dshm" pod "ee6148fe-e516-492b-b5a7-755a75e82c58" (UID: "ee6148fe-e516-492b-b5a7-755a75e82c58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:26.397921 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.397831 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6148fe-e516-492b-b5a7-755a75e82c58-kube-api-access-qmfqx" pod "ee6148fe-e516-492b-b5a7-755a75e82c58" (UID: "ee6148fe-e516-492b-b5a7-755a75e82c58"). InnerVolumeSpecName "kube-api-access-qmfqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:26.497056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.497021 7570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6148fe-e516-492b-b5a7-755a75e82c58-tls-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:26.497056 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.497052 7570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-dshm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:26.497286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.497064 7570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6148fe-e516-492b-b5a7-755a75e82c58-model-cache\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:26.497286 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:26.497079 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmfqx\" (UniqueName: \"kubernetes.io/projected/ee6148fe-e516-492b-b5a7-755a75e82c58-kube-api-access-qmfqx\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:27.330737 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.330688 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl" Oct 30 23:59:27.386454 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.386422 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl"] Oct 30 23:59:27.390603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.390577 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-e15304/distinct-2-w6-e15304-kserve-58ff5bf4cd-ftdpl"] Oct 30 23:59:27.614218 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.614148 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd"] Oct 30 23:59:27.616717 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616698 7570 cpu_manager.go:414] "containerMap: removing container" podUID="dec6f9e3-832c-422c-8172-123b52f2e2e5" containerName="revoke-keys" Oct 30 23:59:27.616778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616721 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec6f9e3-832c-422c-8172-123b52f2e2e5" containerName="revoke-keys" Oct 30 23:59:27.616778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616746 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e0f6ac75-7dd2-44be-8b54-92ff018b8a3d" containerName="payload-processing" Oct 30 23:59:27.616778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616754 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f6ac75-7dd2-44be-8b54-92ff018b8a3d" containerName="payload-processing" Oct 30 23:59:27.616778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616769 7570 cpu_manager.go:414] "containerMap: removing container" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerName="istio-proxy" Oct 30 23:59:27.616778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616774 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerName="istio-proxy" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616793 7570 cpu_manager.go:414] "containerMap: removing container" podUID="1d0b4116-23a4-4074-8dfb-93b7bbf53681" containerName="payload-pre-processing" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616800 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0b4116-23a4-4074-8dfb-93b7bbf53681" containerName="payload-pre-processing" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616809 7570 cpu_manager.go:414] "containerMap: removing container" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616815 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616827 7570 cpu_manager.go:414] "containerMap: removing container" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616832 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616844 7570 cpu_manager.go:414] "containerMap: removing container" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616850 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616860 7570 cpu_manager.go:414] "containerMap: removing container" podUID="6843e367-692b-41f0-a9c6-7704d732f009" containerName="payload-pre-processing" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616865 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6843e367-692b-41f0-a9c6-7704d732f009" containerName="payload-pre-processing" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616886 7570 cpu_manager.go:414] "containerMap: removing container" podUID="08022df4-0b83-43af-9b14-b85ab9f83be3" containerName="payload-processing" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616890 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08022df4-0b83-43af-9b14-b85ab9f83be3" containerName="payload-processing" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616896 7570 cpu_manager.go:414] "containerMap: removing container" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" Oct 30 23:59:27.616905 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616900 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616918 7570 cpu_manager.go:414] "containerMap: removing container" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616923 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616937 7570 cpu_manager.go:414] "containerMap: removing container" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616944 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616957 7570 cpu_manager.go:414] "containerMap: removing container" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.616962 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617058 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf9885e-6fe7-4c33-b91a-60b11bf4b730" containerName="istio-proxy" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617070 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0b4116-23a4-4074-8dfb-93b7bbf53681" containerName="payload-pre-processing" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617075 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617082 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617089 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617094 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6843e367-692b-41f0-a9c6-7704d732f009" containerName="payload-pre-processing" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617102 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="08022df4-0b83-43af-9b14-b85ab9f83be3" containerName="payload-processing" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617109 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" containerName="main" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617117 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b86c0c-3f2f-4f9d-be26-948cb9084373" containerName="istio-proxy" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617123 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617130 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb95af9b-7595-4dd9-839a-e698e2a22c07" containerName="istio-proxy" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617136 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec6f9e3-832c-422c-8172-123b52f2e2e5" containerName="revoke-keys" Oct 30 23:59:27.617267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.617143 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f6ac75-7dd2-44be-8b54-92ff018b8a3d" containerName="payload-processing" Oct 30 23:59:27.623648 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.623632 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:27.630104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.630084 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:59:27.630411 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.630388 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:59:27.648520 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.642816 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd"] Oct 30 23:59:27.718968 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.718936 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpc7t\" (UniqueName: \"kubernetes.io/projected/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-kube-api-access-xpc7t\") pod \"maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd\" (UID: \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:27.719317 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.719257 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd\" (UID: \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:27.719474 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.719442 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd\" (UID: \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:27.820822 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.820790 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpc7t\" (UniqueName: \"kubernetes.io/projected/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-kube-api-access-xpc7t\") pod \"maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd\" (UID: \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:27.820980 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.820969 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd\" (UID: \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:27.821045 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.821030 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd\" (UID: \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:27.821658 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.821631 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd\" (UID: \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:27.824977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.824960 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd\" (UID: \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:27.836263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.836241 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpc7t\" (UniqueName: \"kubernetes.io/projected/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-kube-api-access-xpc7t\") pod \"maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd\" (UID: \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:27.938602 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:27.938542 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:28.200976 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.200892 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp"] Oct 30 23:59:28.202115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.202089 7570 cpu_manager.go:414] "containerMap: removing container" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" Oct 30 23:59:28.202115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.202115 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" Oct 30 23:59:28.202269 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.202140 7570 cpu_manager.go:414] "containerMap: removing container" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" Oct 30 23:59:28.202269 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.202148 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af4419c-fffd-463e-9577-94bad299abc6" containerName="maas-api" Oct 30 23:59:28.202377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.202362 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3d259c-4f6c-4033-b6dd-1059616fe3b4" containerName="maas-api" Oct 30 23:59:28.202419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.202386 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2a6d9c-ec3e-438a-9d6b-db75fa30a903" containerName="maas-api" Oct 30 23:59:28.214162 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.213947 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp"] Oct 30 23:59:28.214162 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.214024 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:28.240875 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.240846 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd"] Oct 30 23:59:28.327781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.327757 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88aa7a16-47cqp\" (UID: \"8cfb49dc-17f9-47ea-a294-246091178077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:28.327947 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.327934 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88aa7a16-47cqp\" (UID: \"8cfb49dc-17f9-47ea-a294-246091178077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:28.328009 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.327998 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvmkj\" (UniqueName: \"kubernetes.io/projected/8cfb49dc-17f9-47ea-a294-246091178077-kube-api-access-pvmkj\") pod \"maas-api-revoke-sub-88aa7a16-47cqp\" (UID: \"8cfb49dc-17f9-47ea-a294-246091178077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:28.342205 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.342175 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" event={"ID":"0512f77a-cc77-4764-9e3b-7d5fd7e6d801","Type":"ContainerStarted","Data":"d2d9555916296b6e44e305e46e0aa704424ab67137aa266551050f184901fd50"} Oct 30 23:59:28.428983 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.428951 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvmkj\" (UniqueName: \"kubernetes.io/projected/8cfb49dc-17f9-47ea-a294-246091178077-kube-api-access-pvmkj\") pod \"maas-api-revoke-sub-88aa7a16-47cqp\" (UID: \"8cfb49dc-17f9-47ea-a294-246091178077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:28.429084 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.429058 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88aa7a16-47cqp\" (UID: \"8cfb49dc-17f9-47ea-a294-246091178077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:28.429250 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.429235 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88aa7a16-47cqp\" (UID: \"8cfb49dc-17f9-47ea-a294-246091178077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:28.429828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.429806 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88aa7a16-47cqp\" (UID: \"8cfb49dc-17f9-47ea-a294-246091178077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:28.431544 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.431525 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88aa7a16-47cqp\" (UID: \"8cfb49dc-17f9-47ea-a294-246091178077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:28.441693 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.441673 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvmkj\" (UniqueName: \"kubernetes.io/projected/8cfb49dc-17f9-47ea-a294-246091178077-kube-api-access-pvmkj\") pod \"maas-api-revoke-sub-88aa7a16-47cqp\" (UID: \"8cfb49dc-17f9-47ea-a294-246091178077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:28.536881 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.536798 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:28.609485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.609434 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6148fe-e516-492b-b5a7-755a75e82c58" path="/var/lib/kubelet/pods/ee6148fe-e516-492b-b5a7-755a75e82c58/volumes" Oct 30 23:59:28.824030 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.824003 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl"] Oct 30 23:59:28.850167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:28.850139 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp"] Oct 30 23:59:28.851078 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:59:28.851035 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfb49dc_17f9_47ea_a294_246091178077.slice/crio-13cfe02ddc95a7b1fec2491085ebfa6250ed3bd2284f88182f1c17d38f29efe3 WatchSource:0}: Error finding container 13cfe02ddc95a7b1fec2491085ebfa6250ed3bd2284f88182f1c17d38f29efe3: Status 404 returned error can't find the container with id 13cfe02ddc95a7b1fec2491085ebfa6250ed3bd2284f88182f1c17d38f29efe3 Oct 30 23:59:29.365884 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:29.365844 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" event={"ID":"0512f77a-cc77-4764-9e3b-7d5fd7e6d801","Type":"ContainerStarted","Data":"ea56ec98a7b7183ec6533b531a5a5c1a2a735c37901b53015239cd43e8e4790b"} Oct 30 23:59:29.367728 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:29.367694 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" event={"ID":"8cfb49dc-17f9-47ea-a294-246091178077","Type":"ContainerStarted","Data":"13cfe02ddc95a7b1fec2491085ebfa6250ed3bd2284f88182f1c17d38f29efe3"} Oct 30 23:59:29.390744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:29.390674 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" podStartSLOduration=2.390655885 podStartE2EDuration="2.390655885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:29.38990469 +0000 UTC m=+2380.908905779" watchObservedRunningTime="2026-10-30 23:59:29.390655885 +0000 UTC m=+2380.909656976" Oct 30 23:59:29.418173 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:29.418133 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:59:29.568981 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:29.568929 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" podUID="7371f1ff-163e-49b2-a4ea-5a9bf3a18215" containerName="payload-pre-processing" containerID="cri-o://e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c" gracePeriod=25 Oct 30 23:59:30.379580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:30.379550 7570 generic.go:356] "Generic (PLEG): container finished" podID="0512f77a-cc77-4764-9e3b-7d5fd7e6d801" containerID="ea56ec98a7b7183ec6533b531a5a5c1a2a735c37901b53015239cd43e8e4790b" exitCode=0 Oct 30 23:59:30.380027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:30.379629 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" event={"ID":"0512f77a-cc77-4764-9e3b-7d5fd7e6d801","Type":"ContainerDied","Data":"ea56ec98a7b7183ec6533b531a5a5c1a2a735c37901b53015239cd43e8e4790b"} Oct 30 23:59:30.381353 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:30.381332 7570 generic.go:356] "Generic (PLEG): container finished" podID="8cfb49dc-17f9-47ea-a294-246091178077" containerID="e4860eb07968b5d32b9ae757e4cedbd8eba31fdeaaf9615e65b7f3ce75b3253e" exitCode=0 Oct 30 23:59:30.381482 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:30.381412 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" event={"ID":"8cfb49dc-17f9-47ea-a294-246091178077","Type":"ContainerDied","Data":"e4860eb07968b5d32b9ae757e4cedbd8eba31fdeaaf9615e65b7f3ce75b3253e"} Oct 30 23:59:30.639066 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:30.638995 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz"] Oct 30 23:59:30.639212 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:30.639178 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" containerID="cri-o://82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f" gracePeriod=30 Oct 30 23:59:31.595476 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.595436 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:31.653003 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.652942 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:31.697491 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.697451 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-openshift-ca\") pod \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\" (UID: \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\") " Oct 30 23:59:31.697640 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.697509 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-kube-api-access-xpc7t\" (UniqueName: \"kubernetes.io/projected/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-kube-api-access-xpc7t\") pod \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\" (UID: \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\") " Oct 30 23:59:31.697773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.697757 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-xks-ca\") pod \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\" (UID: \"0512f77a-cc77-4764-9e3b-7d5fd7e6d801\") " Oct 30 23:59:31.698840 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.698814 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-openshift-ca" pod "0512f77a-cc77-4764-9e3b-7d5fd7e6d801" (UID: "0512f77a-cc77-4764-9e3b-7d5fd7e6d801"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:31.703163 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.703137 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-kube-api-access-xpc7t" pod "0512f77a-cc77-4764-9e3b-7d5fd7e6d801" (UID: "0512f77a-cc77-4764-9e3b-7d5fd7e6d801"). InnerVolumeSpecName "kube-api-access-xpc7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:31.704771 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.704743 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-xks-ca" pod "0512f77a-cc77-4764-9e3b-7d5fd7e6d801" (UID: "0512f77a-cc77-4764-9e3b-7d5fd7e6d801"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:31.798928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.798897 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-xks-ca\") pod \"8cfb49dc-17f9-47ea-a294-246091178077\" (UID: \"8cfb49dc-17f9-47ea-a294-246091178077\") " Oct 30 23:59:31.799129 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.799112 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-openshift-ca\") pod \"8cfb49dc-17f9-47ea-a294-246091178077\" (UID: \"8cfb49dc-17f9-47ea-a294-246091178077\") " Oct 30 23:59:31.799194 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.799168 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfb49dc-17f9-47ea-a294-246091178077-kube-api-access-pvmkj\" (UniqueName: \"kubernetes.io/projected/8cfb49dc-17f9-47ea-a294-246091178077-kube-api-access-pvmkj\") pod \"8cfb49dc-17f9-47ea-a294-246091178077\" (UID: \"8cfb49dc-17f9-47ea-a294-246091178077\") " Oct 30 23:59:31.799584 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.799555 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-openshift-ca" pod "8cfb49dc-17f9-47ea-a294-246091178077" (UID: "8cfb49dc-17f9-47ea-a294-246091178077"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:31.800196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.800172 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:31.800241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.800207 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:31.800241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.800224 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:31.800241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.800238 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpc7t\" (UniqueName: \"kubernetes.io/projected/0512f77a-cc77-4764-9e3b-7d5fd7e6d801-kube-api-access-xpc7t\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:31.801310 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.801287 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-xks-ca" pod "8cfb49dc-17f9-47ea-a294-246091178077" (UID: "8cfb49dc-17f9-47ea-a294-246091178077"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:31.801405 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.801385 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfb49dc-17f9-47ea-a294-246091178077-kube-api-access-pvmkj" pod "8cfb49dc-17f9-47ea-a294-246091178077" (UID: "8cfb49dc-17f9-47ea-a294-246091178077"). InnerVolumeSpecName "kube-api-access-pvmkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:31.901448 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.901418 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvmkj\" (UniqueName: \"kubernetes.io/projected/8cfb49dc-17f9-47ea-a294-246091178077-kube-api-access-pvmkj\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:31.901448 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:31.901447 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfb49dc-17f9-47ea-a294-246091178077-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:32.089711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.089678 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp"] Oct 30 23:59:32.089910 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.089887 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" containerID="cri-o://ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920" gracePeriod=30 Oct 30 23:59:32.410090 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.410001 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" Oct 30 23:59:32.410090 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.410049 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd" event={"ID":"0512f77a-cc77-4764-9e3b-7d5fd7e6d801","Type":"ContainerDied","Data":"d2d9555916296b6e44e305e46e0aa704424ab67137aa266551050f184901fd50"} Oct 30 23:59:32.410434 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.410410 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2d9555916296b6e44e305e46e0aa704424ab67137aa266551050f184901fd50" Oct 30 23:59:32.411744 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.411719 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" Oct 30 23:59:32.411872 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.411733 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp" event={"ID":"8cfb49dc-17f9-47ea-a294-246091178077","Type":"ContainerDied","Data":"13cfe02ddc95a7b1fec2491085ebfa6250ed3bd2284f88182f1c17d38f29efe3"} Oct 30 23:59:32.411872 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.411794 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13cfe02ddc95a7b1fec2491085ebfa6250ed3bd2284f88182f1c17d38f29efe3" Oct 30 23:59:32.562677 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.562650 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw"] Oct 30 23:59:32.563373 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.563360 7570 cpu_manager.go:414] "containerMap: removing container" podUID="0512f77a-cc77-4764-9e3b-7d5fd7e6d801" containerName="revoke-keys" Oct 30 23:59:32.563419 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.563375 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0512f77a-cc77-4764-9e3b-7d5fd7e6d801" containerName="revoke-keys" Oct 30 23:59:32.563492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.563418 7570 cpu_manager.go:414] "containerMap: removing container" podUID="8cfb49dc-17f9-47ea-a294-246091178077" containerName="revoke-keys" Oct 30 23:59:32.563492 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.563424 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfb49dc-17f9-47ea-a294-246091178077" containerName="revoke-keys" Oct 30 23:59:32.563563 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.563557 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfb49dc-17f9-47ea-a294-246091178077" containerName="revoke-keys" Oct 30 23:59:32.563603 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.563572 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="0512f77a-cc77-4764-9e3b-7d5fd7e6d801" containerName="revoke-keys" Oct 30 23:59:32.568689 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.568661 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:32.576105 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.576078 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:59:32.578066 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.578042 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:59:32.582533 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.582509 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw"] Oct 30 23:59:32.716186 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.716106 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw\" (UID: \"506e0a2c-7d48-45e8-981c-d58c4265ff83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:32.716577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.716312 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw\" (UID: \"506e0a2c-7d48-45e8-981c-d58c4265ff83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:32.716577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.716562 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87fq\" (UniqueName: \"kubernetes.io/projected/506e0a2c-7d48-45e8-981c-d58c4265ff83-kube-api-access-c87fq\") pod \"maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw\" (UID: \"506e0a2c-7d48-45e8-981c-d58c4265ff83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:32.818401 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.818370 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw\" (UID: \"506e0a2c-7d48-45e8-981c-d58c4265ff83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:32.818621 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.818602 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87fq\" (UniqueName: \"kubernetes.io/projected/506e0a2c-7d48-45e8-981c-d58c4265ff83-kube-api-access-c87fq\") pod \"maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw\" (UID: \"506e0a2c-7d48-45e8-981c-d58c4265ff83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:32.818795 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.818763 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw\" (UID: \"506e0a2c-7d48-45e8-981c-d58c4265ff83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:32.819103 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.819074 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw\" (UID: \"506e0a2c-7d48-45e8-981c-d58c4265ff83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:32.821271 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.821250 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw\" (UID: \"506e0a2c-7d48-45e8-981c-d58c4265ff83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:32.832537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.832516 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87fq\" (UniqueName: \"kubernetes.io/projected/506e0a2c-7d48-45e8-981c-d58c4265ff83-kube-api-access-c87fq\") pod \"maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw\" (UID: \"506e0a2c-7d48-45e8-981c-d58c4265ff83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:32.892289 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:32.892268 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:33.137351 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:33.137319 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" podUID="7371f1ff-163e-49b2-a4ea-5a9bf3a18215" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 23:59:33.155272 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:33.155231 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" podUID="522aa71b-47e4-44b1-825c-bf81305d21b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 23:59:33.186059 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:59:33.186026 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506e0a2c_7d48_45e8_981c_d58c4265ff83.slice/crio-0051b1480aad1b93dbd78b074e1a6069db70dcb71d4026cbcea88d1d9c7ffe6f WatchSource:0}: Error finding container 0051b1480aad1b93dbd78b074e1a6069db70dcb71d4026cbcea88d1d9c7ffe6f: Status 404 returned error can't find the container with id 0051b1480aad1b93dbd78b074e1a6069db70dcb71d4026cbcea88d1d9c7ffe6f Oct 30 23:59:33.187944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:33.187922 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw"] Oct 30 23:59:33.426863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:33.426821 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" event={"ID":"506e0a2c-7d48-45e8-981c-d58c4265ff83","Type":"ContainerStarted","Data":"0051b1480aad1b93dbd78b074e1a6069db70dcb71d4026cbcea88d1d9c7ffe6f"} Oct 30 23:59:33.825027 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:33.824932 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" podUID="a4acfbb3-3ee2-4890-984a-2fc8a796bece" containerName="payload-processing" containerID="cri-o://3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821" gracePeriod=25 Oct 30 23:59:34.067568 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.067503 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" containerID="cri-o://3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65" gracePeriod=45 Oct 30 23:59:34.241747 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.241713 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:59:34.344037 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.343937 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4acfbb3-3ee2-4890-984a-2fc8a796bece-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4acfbb3-3ee2-4890-984a-2fc8a796bece-plugins-config-volume\") pod \"a4acfbb3-3ee2-4890-984a-2fc8a796bece\" (UID: \"a4acfbb3-3ee2-4890-984a-2fc8a796bece\") " Oct 30 23:59:34.344211 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.344191 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4acfbb3-3ee2-4890-984a-2fc8a796bece-kube-api-access-b7chn\" (UniqueName: \"kubernetes.io/projected/a4acfbb3-3ee2-4890-984a-2fc8a796bece-kube-api-access-b7chn\") pod \"a4acfbb3-3ee2-4890-984a-2fc8a796bece\" (UID: \"a4acfbb3-3ee2-4890-984a-2fc8a796bece\") " Oct 30 23:59:34.344350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.344323 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4acfbb3-3ee2-4890-984a-2fc8a796bece-plugins-config-volume" pod "a4acfbb3-3ee2-4890-984a-2fc8a796bece" (UID: "a4acfbb3-3ee2-4890-984a-2fc8a796bece"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:34.344722 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.344705 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4acfbb3-3ee2-4890-984a-2fc8a796bece-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:34.346842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.346819 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4acfbb3-3ee2-4890-984a-2fc8a796bece-kube-api-access-b7chn" pod "a4acfbb3-3ee2-4890-984a-2fc8a796bece" (UID: "a4acfbb3-3ee2-4890-984a-2fc8a796bece"). InnerVolumeSpecName "kube-api-access-b7chn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:34.382243 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.382207 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:59:34.439971 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.439933 7570 generic.go:356] "Generic (PLEG): container finished" podID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerID="3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65" exitCode=0 Oct 30 23:59:34.440134 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.439989 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" Oct 30 23:59:34.440134 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.440035 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" event={"ID":"00b04bdb-908f-4ced-b3d2-1627d96513b6","Type":"ContainerDied","Data":"3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65"} Oct 30 23:59:34.440134 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.440070 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" event={"ID":"00b04bdb-908f-4ced-b3d2-1627d96513b6","Type":"ContainerDied","Data":"a3db04a3b132ee5a580c3cf71cf87558dffcbf9a52c618d238d14851cecea71f"} Oct 30 23:59:34.440134 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.440092 7570 scope.go:122] "RemoveContainer" containerID="3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65" Oct 30 23:59:34.443362 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.443336 7570 generic.go:356] "Generic (PLEG): container finished" podID="506e0a2c-7d48-45e8-981c-d58c4265ff83" containerID="1b2b11e55ea179c0a86d92b0ea3376b0e745720c9f4ce8971e68e8f35f75dfab" exitCode=0 Oct 30 23:59:34.443590 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.443377 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" event={"ID":"506e0a2c-7d48-45e8-981c-d58c4265ff83","Type":"ContainerDied","Data":"1b2b11e55ea179c0a86d92b0ea3376b0e745720c9f4ce8971e68e8f35f75dfab"} Oct 30 23:59:34.446652 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.446629 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7chn\" (UniqueName: \"kubernetes.io/projected/a4acfbb3-3ee2-4890-984a-2fc8a796bece-kube-api-access-b7chn\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:34.472965 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.472936 7570 generic.go:356] "Generic (PLEG): container finished" podID="a4acfbb3-3ee2-4890-984a-2fc8a796bece" containerID="3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821" exitCode=0 Oct 30 23:59:34.473107 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.473091 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" event={"ID":"a4acfbb3-3ee2-4890-984a-2fc8a796bece","Type":"ContainerDied","Data":"3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821"} Oct 30 23:59:34.473176 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.473142 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" event={"ID":"a4acfbb3-3ee2-4890-984a-2fc8a796bece","Type":"ContainerDied","Data":"4fe5703ab82627b5845517038cf50e6b42c7ac86c43b503bddefc6a123c89810"} Oct 30 23:59:34.473229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.473221 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl" Oct 30 23:59:34.481643 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.481623 7570 scope.go:122] "RemoveContainer" containerID="3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65" Oct 30 23:59:34.485088 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:34.484633 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65\": container with ID starting with 3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65 not found: ID does not exist" containerID="3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65" Oct 30 23:59:34.485088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.484671 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65"} err="failed to get container status \"3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65\": rpc error: code = NotFound desc = could not find container \"3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65\": container with ID starting with 3a8676e2bb8fba24312664d9c0461c24b8676c0ae0a92c11ca2d48e5db3daa65 not found: ID does not exist" Oct 30 23:59:34.485088 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.484694 7570 scope.go:122] "RemoveContainer" containerID="3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821" Oct 30 23:59:34.502447 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.502370 7570 scope.go:122] "RemoveContainer" containerID="3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821" Oct 30 23:59:34.502814 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:34.502792 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821\": container with ID starting with 3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821 not found: ID does not exist" containerID="3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821" Oct 30 23:59:34.502887 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.502824 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821"} err="failed to get container status \"3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821\": rpc error: code = NotFound desc = could not find container \"3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821\": container with ID starting with 3d2c4d4bb53970004e4bcac89a76b34f5413d8a2b8ad45443d64de99d585a821 not found: ID does not exist" Oct 30 23:59:34.529130 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.529101 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl"] Oct 30 23:59:34.543935 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.543907 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f0817c-69bf459597-c4kvl"] Oct 30 23:59:34.547580 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.547550 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-dshm\") pod \"00b04bdb-908f-4ced-b3d2-1627d96513b6\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " Oct 30 23:59:34.547752 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.547735 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-home\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-home\") pod \"00b04bdb-908f-4ced-b3d2-1627d96513b6\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " Oct 30 23:59:34.547842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.547828 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00b04bdb-908f-4ced-b3d2-1627d96513b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/00b04bdb-908f-4ced-b3d2-1627d96513b6-tls-certs\") pod \"00b04bdb-908f-4ced-b3d2-1627d96513b6\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " Oct 30 23:59:34.547906 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.547882 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b04bdb-908f-4ced-b3d2-1627d96513b6-kube-api-access-xw6ps\" (UniqueName: \"kubernetes.io/projected/00b04bdb-908f-4ced-b3d2-1627d96513b6-kube-api-access-xw6ps\") pod \"00b04bdb-908f-4ced-b3d2-1627d96513b6\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " Oct 30 23:59:34.547960 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.547941 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-model-cache\") pod \"00b04bdb-908f-4ced-b3d2-1627d96513b6\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " Oct 30 23:59:34.548017 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.547981 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-tmp-dir\") pod \"00b04bdb-908f-4ced-b3d2-1627d96513b6\" (UID: \"00b04bdb-908f-4ced-b3d2-1627d96513b6\") " Oct 30 23:59:34.549661 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.549638 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-tmp-dir" pod "00b04bdb-908f-4ced-b3d2-1627d96513b6" (UID: "00b04bdb-908f-4ced-b3d2-1627d96513b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:34.550496 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.550434 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-model-cache" pod "00b04bdb-908f-4ced-b3d2-1627d96513b6" (UID: "00b04bdb-908f-4ced-b3d2-1627d96513b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:34.550957 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.550931 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-home" pod "00b04bdb-908f-4ced-b3d2-1627d96513b6" (UID: "00b04bdb-908f-4ced-b3d2-1627d96513b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:34.552868 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.552847 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00b04bdb-908f-4ced-b3d2-1627d96513b6-tls-certs" pod "00b04bdb-908f-4ced-b3d2-1627d96513b6" (UID: "00b04bdb-908f-4ced-b3d2-1627d96513b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:34.553146 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.553129 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-dshm" pod "00b04bdb-908f-4ced-b3d2-1627d96513b6" (UID: "00b04bdb-908f-4ced-b3d2-1627d96513b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:34.553354 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.553340 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b04bdb-908f-4ced-b3d2-1627d96513b6-kube-api-access-xw6ps" pod "00b04bdb-908f-4ced-b3d2-1627d96513b6" (UID: "00b04bdb-908f-4ced-b3d2-1627d96513b6"). InnerVolumeSpecName "kube-api-access-xw6ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:34.604312 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.604260 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4acfbb3-3ee2-4890-984a-2fc8a796bece" path="/var/lib/kubelet/pods/a4acfbb3-3ee2-4890-984a-2fc8a796bece/volumes" Oct 30 23:59:34.650241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.650208 7570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-dshm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:34.650241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.650239 7570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-home\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:34.650241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.650248 7570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00b04bdb-908f-4ced-b3d2-1627d96513b6-tls-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:34.650485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.650258 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw6ps\" (UniqueName: \"kubernetes.io/projected/00b04bdb-908f-4ced-b3d2-1627d96513b6-kube-api-access-xw6ps\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:34.650485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.650269 7570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-model-cache\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:34.650485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.650300 7570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b04bdb-908f-4ced-b3d2-1627d96513b6-tmp-dir\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:34.761880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.761850 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt"] Oct 30 23:59:34.764563 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:34.764542 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt"] Oct 30 23:59:35.359474 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.359420 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/simulator-w1-611ef3-kserve-8455bfc968-b9llt" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:59:35.778558 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.778535 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:35.880004 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.879966 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506e0a2c-7d48-45e8-981c-d58c4265ff83-kube-api-access-c87fq\" (UniqueName: \"kubernetes.io/projected/506e0a2c-7d48-45e8-981c-d58c4265ff83-kube-api-access-c87fq\") pod \"506e0a2c-7d48-45e8-981c-d58c4265ff83\" (UID: \"506e0a2c-7d48-45e8-981c-d58c4265ff83\") " Oct 30 23:59:35.887553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.880127 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-openshift-ca\") pod \"506e0a2c-7d48-45e8-981c-d58c4265ff83\" (UID: \"506e0a2c-7d48-45e8-981c-d58c4265ff83\") " Oct 30 23:59:35.887553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.880195 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-xks-ca\") pod \"506e0a2c-7d48-45e8-981c-d58c4265ff83\" (UID: \"506e0a2c-7d48-45e8-981c-d58c4265ff83\") " Oct 30 23:59:35.887553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.880895 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-openshift-ca" pod "506e0a2c-7d48-45e8-981c-d58c4265ff83" (UID: "506e0a2c-7d48-45e8-981c-d58c4265ff83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:35.887553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.881064 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:35.887553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.883820 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506e0a2c-7d48-45e8-981c-d58c4265ff83-kube-api-access-c87fq" pod "506e0a2c-7d48-45e8-981c-d58c4265ff83" (UID: "506e0a2c-7d48-45e8-981c-d58c4265ff83"). InnerVolumeSpecName "kube-api-access-c87fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:35.887553 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.885805 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-xks-ca" pod "506e0a2c-7d48-45e8-981c-d58c4265ff83" (UID: "506e0a2c-7d48-45e8-981c-d58c4265ff83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:35.942656 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.942633 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:59:35.981659 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.981627 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-podinfo\") pod \"48390cfe-87a3-4b4e-9986-8adec60f6349\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " Oct 30 23:59:35.981790 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.981699 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-token\" (UniqueName: \"kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-token\") pod \"48390cfe-87a3-4b4e-9986-8adec60f6349\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " Oct 30 23:59:35.981862 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.981802 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48390cfe-87a3-4b4e-9986-8adec60f6349-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48390cfe-87a3-4b4e-9986-8adec60f6349-istiod-ca-cert\") pod \"48390cfe-87a3-4b4e-9986-8adec60f6349\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " Oct 30 23:59:35.981862 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.981842 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-data\") pod \"48390cfe-87a3-4b4e-9986-8adec60f6349\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " Oct 30 23:59:35.981959 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.981876 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-envoy\") pod \"48390cfe-87a3-4b4e-9986-8adec60f6349\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " Oct 30 23:59:35.982166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.982048 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-credential-socket\") pod \"48390cfe-87a3-4b4e-9986-8adec60f6349\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " Oct 30 23:59:35.982166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.982106 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-data" pod "48390cfe-87a3-4b4e-9986-8adec60f6349" (UID: "48390cfe-87a3-4b4e-9986-8adec60f6349"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:35.982166 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.982137 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-socket\") pod \"48390cfe-87a3-4b4e-9986-8adec60f6349\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " Oct 30 23:59:35.982347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.982178 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-certs\") pod \"48390cfe-87a3-4b4e-9986-8adec60f6349\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " Oct 30 23:59:35.982347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.982305 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-kube-api-access-q87ln\" (UniqueName: \"kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-kube-api-access-q87ln\") pod \"48390cfe-87a3-4b4e-9986-8adec60f6349\" (UID: \"48390cfe-87a3-4b4e-9986-8adec60f6349\") " Oct 30 23:59:35.983021 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.982775 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48390cfe-87a3-4b4e-9986-8adec60f6349-istiod-ca-cert" pod "48390cfe-87a3-4b4e-9986-8adec60f6349" (UID: "48390cfe-87a3-4b4e-9986-8adec60f6349"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:35.983119 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.983072 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-socket" pod "48390cfe-87a3-4b4e-9986-8adec60f6349" (UID: "48390cfe-87a3-4b4e-9986-8adec60f6349"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:35.983171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.983129 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c87fq\" (UniqueName: \"kubernetes.io/projected/506e0a2c-7d48-45e8-981c-d58c4265ff83-kube-api-access-c87fq\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:35.983171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.983150 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506e0a2c-7d48-45e8-981c-d58c4265ff83-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:35.983171 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.983165 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48390cfe-87a3-4b4e-9986-8adec60f6349-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:35.983519 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.983178 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:35.983519 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.983252 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-certs" pod "48390cfe-87a3-4b4e-9986-8adec60f6349" (UID: "48390cfe-87a3-4b4e-9986-8adec60f6349"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:35.983519 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.983327 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-credential-socket" pod "48390cfe-87a3-4b4e-9986-8adec60f6349" (UID: "48390cfe-87a3-4b4e-9986-8adec60f6349"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:35.984446 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.984403 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-podinfo" pod "48390cfe-87a3-4b4e-9986-8adec60f6349" (UID: "48390cfe-87a3-4b4e-9986-8adec60f6349"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:59:35.984576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.984511 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-token" pod "48390cfe-87a3-4b4e-9986-8adec60f6349" (UID: "48390cfe-87a3-4b4e-9986-8adec60f6349"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:35.985494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.985429 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-kube-api-access-q87ln" pod "48390cfe-87a3-4b4e-9986-8adec60f6349" (UID: "48390cfe-87a3-4b4e-9986-8adec60f6349"). InnerVolumeSpecName "kube-api-access-q87ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:35.985738 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:35.985711 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-envoy" pod "48390cfe-87a3-4b4e-9986-8adec60f6349" (UID: "48390cfe-87a3-4b4e-9986-8adec60f6349"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:36.085044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.085011 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:36.085044 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.085045 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:36.085229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.085061 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:36.085229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.085078 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:36.085229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.085090 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48390cfe-87a3-4b4e-9986-8adec60f6349-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:36.085229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.085102 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q87ln\" (UniqueName: \"kubernetes.io/projected/48390cfe-87a3-4b4e-9986-8adec60f6349-kube-api-access-q87ln\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:36.085229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.085116 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48390cfe-87a3-4b4e-9986-8adec60f6349-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:36.495406 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.495373 7570 generic.go:356] "Generic (PLEG): container finished" podID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerID="82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f" exitCode=0 Oct 30 23:59:36.495852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.495417 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" Oct 30 23:59:36.495852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.495452 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" event={"ID":"48390cfe-87a3-4b4e-9986-8adec60f6349","Type":"ContainerDied","Data":"82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f"} Oct 30 23:59:36.495852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.495560 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz" event={"ID":"48390cfe-87a3-4b4e-9986-8adec60f6349","Type":"ContainerDied","Data":"abcae9ccc790e314436463e92fd791146e561b7a1fa286837d5991a94913d592"} Oct 30 23:59:36.495852 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.495598 7570 scope.go:122] "RemoveContainer" containerID="82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f" Oct 30 23:59:36.497241 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.497227 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" Oct 30 23:59:36.497301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.497237 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw" event={"ID":"506e0a2c-7d48-45e8-981c-d58c4265ff83","Type":"ContainerDied","Data":"0051b1480aad1b93dbd78b074e1a6069db70dcb71d4026cbcea88d1d9c7ffe6f"} Oct 30 23:59:36.497301 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.497289 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0051b1480aad1b93dbd78b074e1a6069db70dcb71d4026cbcea88d1d9c7ffe6f" Oct 30 23:59:36.509191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.509174 7570 scope.go:122] "RemoveContainer" containerID="8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738" Oct 30 23:59:36.521586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.521566 7570 scope.go:122] "RemoveContainer" containerID="82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f" Oct 30 23:59:36.521887 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:36.521870 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f\": container with ID starting with 82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f not found: ID does not exist" containerID="82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f" Oct 30 23:59:36.521928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.521895 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f"} err="failed to get container status \"82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f\": rpc error: code = NotFound desc = could not find container \"82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f\": container with ID starting with 82597d5398a600690987d379d3535da1969dea2f1faebf4d17f26209d7775a7f not found: ID does not exist" Oct 30 23:59:36.521928 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.521911 7570 scope.go:122] "RemoveContainer" containerID="8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738" Oct 30 23:59:36.522185 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:36.522163 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738\": container with ID starting with 8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738 not found: ID does not exist" containerID="8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738" Oct 30 23:59:36.522238 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.522194 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738"} err="failed to get container status \"8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738\": rpc error: code = NotFound desc = could not find container \"8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738\": container with ID starting with 8a34fc3901f5bf91bbc666dea02291c817af7d6d433d516ededc17d0eea86738 not found: ID does not exist" Oct 30 23:59:36.532126 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.532105 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz"] Oct 30 23:59:36.537557 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.537536 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6c5084-openshift-default-6f596cc46f-f7ffz"] Oct 30 23:59:36.613819 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.613792 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" path="/var/lib/kubelet/pods/00b04bdb-908f-4ced-b3d2-1627d96513b6/volumes" Oct 30 23:59:36.614270 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.614258 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" path="/var/lib/kubelet/pods/48390cfe-87a3-4b4e-9986-8adec60f6349/volumes" Oct 30 23:59:36.951531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.951489 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q"] Oct 30 23:59:36.952195 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952176 7570 cpu_manager.go:414] "containerMap: removing container" podUID="506e0a2c-7d48-45e8-981c-d58c4265ff83" containerName="revoke-keys" Oct 30 23:59:36.952298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952198 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506e0a2c-7d48-45e8-981c-d58c4265ff83" containerName="revoke-keys" Oct 30 23:59:36.952298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952247 7570 cpu_manager.go:414] "containerMap: removing container" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" Oct 30 23:59:36.952298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952255 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" Oct 30 23:59:36.952298 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952293 7570 cpu_manager.go:414] "containerMap: removing container" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" Oct 30 23:59:36.952532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952302 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" Oct 30 23:59:36.952532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952314 7570 cpu_manager.go:414] "containerMap: removing container" podUID="a4acfbb3-3ee2-4890-984a-2fc8a796bece" containerName="payload-processing" Oct 30 23:59:36.952532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952322 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4acfbb3-3ee2-4890-984a-2fc8a796bece" containerName="payload-processing" Oct 30 23:59:36.952532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952355 7570 cpu_manager.go:414] "containerMap: removing container" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" Oct 30 23:59:36.952532 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952362 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" Oct 30 23:59:36.952751 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952548 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="506e0a2c-7d48-45e8-981c-d58c4265ff83" containerName="revoke-keys" Oct 30 23:59:36.952751 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952569 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" Oct 30 23:59:36.952751 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952584 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="48390cfe-87a3-4b4e-9986-8adec60f6349" containerName="istio-proxy" Oct 30 23:59:36.952751 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952592 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4acfbb3-3ee2-4890-984a-2fc8a796bece" containerName="payload-processing" Oct 30 23:59:36.952751 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.952607 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b04bdb-908f-4ced-b3d2-1627d96513b6" containerName="main" Oct 30 23:59:36.957729 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.957710 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:36.963163 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.963142 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:59:36.963573 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.963557 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:59:36.969666 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.968388 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q"] Oct 30 23:59:36.996025 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.995998 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drvd\" (UniqueName: \"kubernetes.io/projected/6530eea0-8758-41fb-8249-a151d46c8ebf-kube-api-access-4drvd\") pod \"maas-api-revoke-sub-48ca4c26-2hx6q\" (UID: \"6530eea0-8758-41fb-8249-a151d46c8ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:36.996172 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.996152 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ca4c26-2hx6q\" (UID: \"6530eea0-8758-41fb-8249-a151d46c8ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:36.996228 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:36.996208 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ca4c26-2hx6q\" (UID: \"6530eea0-8758-41fb-8249-a151d46c8ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:37.098083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.098058 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ca4c26-2hx6q\" (UID: \"6530eea0-8758-41fb-8249-a151d46c8ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:37.098198 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.098091 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ca4c26-2hx6q\" (UID: \"6530eea0-8758-41fb-8249-a151d46c8ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:37.098315 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.098291 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drvd\" (UniqueName: \"kubernetes.io/projected/6530eea0-8758-41fb-8249-a151d46c8ebf-kube-api-access-4drvd\") pod \"maas-api-revoke-sub-48ca4c26-2hx6q\" (UID: \"6530eea0-8758-41fb-8249-a151d46c8ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:37.098909 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.098886 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ca4c26-2hx6q\" (UID: \"6530eea0-8758-41fb-8249-a151d46c8ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:37.100863 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.100841 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ca4c26-2hx6q\" (UID: \"6530eea0-8758-41fb-8249-a151d46c8ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:37.111748 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.111723 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drvd\" (UniqueName: \"kubernetes.io/projected/6530eea0-8758-41fb-8249-a151d46c8ebf-kube-api-access-4drvd\") pod \"maas-api-revoke-sub-48ca4c26-2hx6q\" (UID: \"6530eea0-8758-41fb-8249-a151d46c8ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:37.283525 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.282325 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:37.391715 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.391690 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:59:37.508974 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.508896 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-data\") pod \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " Oct 30 23:59:37.509414 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.508998 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-certs\") pod \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " Oct 30 23:59:37.509414 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.509035 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-socket\") pod \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " Oct 30 23:59:37.509414 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.509100 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istiod-ca-cert\") pod \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " Oct 30 23:59:37.509414 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.509140 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-token\" (UniqueName: \"kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-token\") pod \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " Oct 30 23:59:37.509414 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.509239 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-envoy\") pod \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " Oct 30 23:59:37.509414 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.509345 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-podinfo\") pod \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " Oct 30 23:59:37.509414 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.509349 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-certs" pod "f3856a78-100e-48e0-aa1e-9a04bb10c99d" (UID: "f3856a78-100e-48e0-aa1e-9a04bb10c99d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:37.509781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.509405 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-socket" pod "f3856a78-100e-48e0-aa1e-9a04bb10c99d" (UID: "f3856a78-100e-48e0-aa1e-9a04bb10c99d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:37.509781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.509443 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-credential-socket\") pod \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " Oct 30 23:59:37.509781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.509543 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-kube-api-access-cwsfc\" (UniqueName: \"kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-kube-api-access-cwsfc\") pod \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\" (UID: \"f3856a78-100e-48e0-aa1e-9a04bb10c99d\") " Oct 30 23:59:37.509781 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.509744 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-credential-socket" pod "f3856a78-100e-48e0-aa1e-9a04bb10c99d" (UID: "f3856a78-100e-48e0-aa1e-9a04bb10c99d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:37.510029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.509836 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istiod-ca-cert" pod "f3856a78-100e-48e0-aa1e-9a04bb10c99d" (UID: "f3856a78-100e-48e0-aa1e-9a04bb10c99d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:37.510029 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.509999 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-data" pod "f3856a78-100e-48e0-aa1e-9a04bb10c99d" (UID: "f3856a78-100e-48e0-aa1e-9a04bb10c99d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:37.510687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.510659 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:37.510773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.510694 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:37.510773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.510709 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:37.510773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.510723 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:37.510773 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.510738 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:37.512554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.512484 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-kube-api-access-cwsfc" pod "f3856a78-100e-48e0-aa1e-9a04bb10c99d" (UID: "f3856a78-100e-48e0-aa1e-9a04bb10c99d"). InnerVolumeSpecName "kube-api-access-cwsfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:37.512554 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.512519 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-envoy" pod "f3856a78-100e-48e0-aa1e-9a04bb10c99d" (UID: "f3856a78-100e-48e0-aa1e-9a04bb10c99d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:37.512845 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.512818 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-token" pod "f3856a78-100e-48e0-aa1e-9a04bb10c99d" (UID: "f3856a78-100e-48e0-aa1e-9a04bb10c99d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:37.512919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.512904 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-podinfo" pod "f3856a78-100e-48e0-aa1e-9a04bb10c99d" (UID: "f3856a78-100e-48e0-aa1e-9a04bb10c99d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:59:37.513994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.513967 7570 generic.go:356] "Generic (PLEG): container finished" podID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerID="ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920" exitCode=0 Oct 30 23:59:37.514089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.514028 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" Oct 30 23:59:37.514089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.514047 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" event={"ID":"f3856a78-100e-48e0-aa1e-9a04bb10c99d","Type":"ContainerDied","Data":"ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920"} Oct 30 23:59:37.514185 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.514099 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp" event={"ID":"f3856a78-100e-48e0-aa1e-9a04bb10c99d","Type":"ContainerDied","Data":"0233294017b589cf09a602503e5aab60a552d54bde063fde208d73ed4620ade2"} Oct 30 23:59:37.514185 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.514121 7570 scope.go:122] "RemoveContainer" containerID="ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920" Oct 30 23:59:37.526411 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.526397 7570 scope.go:122] "RemoveContainer" containerID="5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870" Oct 30 23:59:37.537183 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.537163 7570 scope.go:122] "RemoveContainer" containerID="ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920" Oct 30 23:59:37.537437 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:37.537420 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920\": container with ID starting with ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920 not found: ID does not exist" containerID="ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920" Oct 30 23:59:37.537494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.537444 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920"} err="failed to get container status \"ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920\": rpc error: code = NotFound desc = could not find container \"ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920\": container with ID starting with ee228bb90162e9edc6bfc597719d5423b227921b7fea86fbe79be9618f149920 not found: ID does not exist" Oct 30 23:59:37.537494 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.537475 7570 scope.go:122] "RemoveContainer" containerID="5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870" Oct 30 23:59:37.537710 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:37.537693 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870\": container with ID starting with 5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870 not found: ID does not exist" containerID="5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870" Oct 30 23:59:37.537755 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.537715 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870"} err="failed to get container status \"5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870\": rpc error: code = NotFound desc = could not find container \"5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870\": container with ID starting with 5b443bda17239db205c0aa802b10361b5ffd8b8430e6a4483d55acfdcd104870 not found: ID does not exist" Oct 30 23:59:37.547283 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.547254 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp"] Oct 30 23:59:37.550622 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.550603 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-wrttp"] Oct 30 23:59:37.612293 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.612269 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwsfc\" (UniqueName: \"kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-kube-api-access-cwsfc\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:37.612293 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.612292 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:37.612429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.612301 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:37.612429 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.612310 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3856a78-100e-48e0-aa1e-9a04bb10c99d-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:37.629375 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:37.629355 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q"] Oct 30 23:59:37.630576 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:59:37.630553 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6530eea0_8758_41fb_8249_a151d46c8ebf.slice/crio-9a4bab9ec4ae949252422294df63654a8048ad07cda3803fca80b19ca989d44e WatchSource:0}: Error finding container 9a4bab9ec4ae949252422294df63654a8048ad07cda3803fca80b19ca989d44e: Status 404 returned error can't find the container with id 9a4bab9ec4ae949252422294df63654a8048ad07cda3803fca80b19ca989d44e Oct 30 23:59:38.525356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:38.525315 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" event={"ID":"6530eea0-8758-41fb-8249-a151d46c8ebf","Type":"ContainerStarted","Data":"a6ef2d1f1857336abefb165a86f274daf6e7a5f5213d48285f752db7f46e54a7"} Oct 30 23:59:38.525733 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:38.525396 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" event={"ID":"6530eea0-8758-41fb-8249-a151d46c8ebf","Type":"ContainerStarted","Data":"9a4bab9ec4ae949252422294df63654a8048ad07cda3803fca80b19ca989d44e"} Oct 30 23:59:38.545950 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:38.545903 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" podStartSLOduration=2.545888788 podStartE2EDuration="2.545888788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:38.544397411 +0000 UTC m=+2390.063398512" watchObservedRunningTime="2026-10-30 23:59:38.545888788 +0000 UTC m=+2390.064889876" Oct 30 23:59:38.604541 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:38.604521 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" path="/var/lib/kubelet/pods/f3856a78-100e-48e0-aa1e-9a04bb10c99d/volumes" Oct 30 23:59:39.538390 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:39.538354 7570 generic.go:356] "Generic (PLEG): container finished" podID="6530eea0-8758-41fb-8249-a151d46c8ebf" containerID="a6ef2d1f1857336abefb165a86f274daf6e7a5f5213d48285f752db7f46e54a7" exitCode=0 Oct 30 23:59:39.538816 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:39.538440 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" event={"ID":"6530eea0-8758-41fb-8249-a151d46c8ebf","Type":"ContainerDied","Data":"a6ef2d1f1857336abefb165a86f274daf6e7a5f5213d48285f752db7f46e54a7"} Oct 30 23:59:40.705564 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:40.705541 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:40.765356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:40.765322 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6530eea0-8758-41fb-8249-a151d46c8ebf-kube-api-access-4drvd\" (UniqueName: \"kubernetes.io/projected/6530eea0-8758-41fb-8249-a151d46c8ebf-kube-api-access-4drvd\") pod \"6530eea0-8758-41fb-8249-a151d46c8ebf\" (UID: \"6530eea0-8758-41fb-8249-a151d46c8ebf\") " Oct 30 23:59:40.765537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:40.765404 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-openshift-ca\") pod \"6530eea0-8758-41fb-8249-a151d46c8ebf\" (UID: \"6530eea0-8758-41fb-8249-a151d46c8ebf\") " Oct 30 23:59:40.765537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:40.765439 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-xks-ca\") pod \"6530eea0-8758-41fb-8249-a151d46c8ebf\" (UID: \"6530eea0-8758-41fb-8249-a151d46c8ebf\") " Oct 30 23:59:40.767923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:40.767891 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-openshift-ca" pod "6530eea0-8758-41fb-8249-a151d46c8ebf" (UID: "6530eea0-8758-41fb-8249-a151d46c8ebf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:40.768309 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:40.768290 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6530eea0-8758-41fb-8249-a151d46c8ebf-kube-api-access-4drvd" pod "6530eea0-8758-41fb-8249-a151d46c8ebf" (UID: "6530eea0-8758-41fb-8249-a151d46c8ebf"). InnerVolumeSpecName "kube-api-access-4drvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:40.770326 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:40.770292 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-xks-ca" pod "6530eea0-8758-41fb-8249-a151d46c8ebf" (UID: "6530eea0-8758-41fb-8249-a151d46c8ebf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:40.872290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:40.872252 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4drvd\" (UniqueName: \"kubernetes.io/projected/6530eea0-8758-41fb-8249-a151d46c8ebf-kube-api-access-4drvd\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:40.872290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:40.872286 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:40.872290 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:40.872297 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6530eea0-8758-41fb-8249-a151d46c8ebf-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:41.325650 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.325567 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb"] Oct 30 23:59:41.561115 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.561081 7570 generic.go:356] "Generic (PLEG): container finished" podID="522aa71b-47e4-44b1-825c-bf81305d21b4" containerID="bf345e06964e0f29e4f25c50b80cc58ae8f87f140017f2f23d4fa69b57312ece" exitCode=0 Oct 30 23:59:41.561500 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.561151 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" event={"ID":"522aa71b-47e4-44b1-825c-bf81305d21b4","Type":"ContainerDied","Data":"bf345e06964e0f29e4f25c50b80cc58ae8f87f140017f2f23d4fa69b57312ece"} Oct 30 23:59:41.563031 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.563013 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" Oct 30 23:59:41.563142 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.563049 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q" event={"ID":"6530eea0-8758-41fb-8249-a151d46c8ebf","Type":"ContainerDied","Data":"9a4bab9ec4ae949252422294df63654a8048ad07cda3803fca80b19ca989d44e"} Oct 30 23:59:41.563142 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.563098 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a4bab9ec4ae949252422294df63654a8048ad07cda3803fca80b19ca989d44e" Oct 30 23:59:41.767828 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.767805 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:59:41.887191 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.887162 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522aa71b-47e4-44b1-825c-bf81305d21b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522aa71b-47e4-44b1-825c-bf81305d21b4-plugins-config-volume\") pod \"522aa71b-47e4-44b1-825c-bf81305d21b4\" (UID: \"522aa71b-47e4-44b1-825c-bf81305d21b4\") " Oct 30 23:59:41.887431 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.887418 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522aa71b-47e4-44b1-825c-bf81305d21b4-kube-api-access-pvzq2\" (UniqueName: \"kubernetes.io/projected/522aa71b-47e4-44b1-825c-bf81305d21b4-kube-api-access-pvzq2\") pod \"522aa71b-47e4-44b1-825c-bf81305d21b4\" (UID: \"522aa71b-47e4-44b1-825c-bf81305d21b4\") " Oct 30 23:59:41.887601 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.887580 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522aa71b-47e4-44b1-825c-bf81305d21b4-plugins-config-volume" pod "522aa71b-47e4-44b1-825c-bf81305d21b4" (UID: "522aa71b-47e4-44b1-825c-bf81305d21b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:41.887900 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.887883 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522aa71b-47e4-44b1-825c-bf81305d21b4-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:41.889694 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.889669 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522aa71b-47e4-44b1-825c-bf81305d21b4-kube-api-access-pvzq2" pod "522aa71b-47e4-44b1-825c-bf81305d21b4" (UID: "522aa71b-47e4-44b1-825c-bf81305d21b4"). InnerVolumeSpecName "kube-api-access-pvzq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:41.989919 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:41.989852 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvzq2\" (UniqueName: \"kubernetes.io/projected/522aa71b-47e4-44b1-825c-bf81305d21b4-kube-api-access-pvzq2\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:42.573613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:42.573581 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" Oct 30 23:59:42.573798 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:42.573619 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp" event={"ID":"522aa71b-47e4-44b1-825c-bf81305d21b4","Type":"ContainerDied","Data":"77a6206811d38439424d8cfd4ae9850942b50cf9cc4e90a5ff525bd300babfd2"} Oct 30 23:59:42.573798 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:42.573682 7570 scope.go:122] "RemoveContainer" containerID="bf345e06964e0f29e4f25c50b80cc58ae8f87f140017f2f23d4fa69b57312ece" Oct 30 23:59:42.603797 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:42.603774 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp"] Oct 30 23:59:42.611237 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:42.611218 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e15304-75dfbc95bc-6w8dp"] Oct 30 23:59:42.734233 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:42.734206 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg"] Oct 30 23:59:43.213663 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:43.213618 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" podUID="7371f1ff-163e-49b2-a4ea-5a9bf3a18215" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 23:59:43.343357 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:43.343323 7570 scope.go:122] "RemoveContainer" containerID="4d5e41c0d0ab89419a20ad13530d6b08ee78346448059542a0475a7805ffd355" Oct 30 23:59:43.708502 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:43.708445 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5"] Oct 30 23:59:43.708775 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:43.708747 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" containerID="cri-o://aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b" gracePeriod=30 Oct 30 23:59:43.980667 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:43.980637 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd"] Oct 30 23:59:43.984216 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:43.984196 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1bc837-5hbvd"] Oct 30 23:59:43.999350 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:43.999263 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:59:44.040387 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.040352 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-maas-api-tls\") pod \"06e6c870-08ba-4b3c-a26d-45397d5ca730\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " Oct 30 23:59:44.040595 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.040455 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e6c870-08ba-4b3c-a26d-45397d5ca730-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e6c870-08ba-4b3c-a26d-45397d5ca730-openshift-service-ca\") pod \"06e6c870-08ba-4b3c-a26d-45397d5ca730\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " Oct 30 23:59:44.040818 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.040792 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-metrics-certs\") pod \"06e6c870-08ba-4b3c-a26d-45397d5ca730\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " Oct 30 23:59:44.040915 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.040840 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e6c870-08ba-4b3c-a26d-45397d5ca730-kube-api-access-z2qwd\" (UniqueName: \"kubernetes.io/projected/06e6c870-08ba-4b3c-a26d-45397d5ca730-kube-api-access-z2qwd\") pod \"06e6c870-08ba-4b3c-a26d-45397d5ca730\" (UID: \"06e6c870-08ba-4b3c-a26d-45397d5ca730\") " Oct 30 23:59:44.041921 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.041897 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e6c870-08ba-4b3c-a26d-45397d5ca730-openshift-service-ca" pod "06e6c870-08ba-4b3c-a26d-45397d5ca730" (UID: "06e6c870-08ba-4b3c-a26d-45397d5ca730"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:44.043152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.043123 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-maas-api-tls" pod "06e6c870-08ba-4b3c-a26d-45397d5ca730" (UID: "06e6c870-08ba-4b3c-a26d-45397d5ca730"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:44.044133 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.044108 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e6c870-08ba-4b3c-a26d-45397d5ca730-kube-api-access-z2qwd" pod "06e6c870-08ba-4b3c-a26d-45397d5ca730" (UID: "06e6c870-08ba-4b3c-a26d-45397d5ca730"). InnerVolumeSpecName "kube-api-access-z2qwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:44.044236 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.044218 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-metrics-certs" pod "06e6c870-08ba-4b3c-a26d-45397d5ca730" (UID: "06e6c870-08ba-4b3c-a26d-45397d5ca730"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:44.169495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.163410 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:44.169495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.163449 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2qwd\" (UniqueName: \"kubernetes.io/projected/06e6c870-08ba-4b3c-a26d-45397d5ca730-kube-api-access-z2qwd\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:44.169495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.163482 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e6c870-08ba-4b3c-a26d-45397d5ca730-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:44.169495 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.163495 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e6c870-08ba-4b3c-a26d-45397d5ca730-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:44.596503 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.596452 7570 generic.go:356] "Generic (PLEG): container finished" podID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerID="aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b" exitCode=0 Oct 30 23:59:44.596944 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.596548 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" Oct 30 23:59:44.603886 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.603856 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0512f77a-cc77-4764-9e3b-7d5fd7e6d801" path="/var/lib/kubelet/pods/0512f77a-cc77-4764-9e3b-7d5fd7e6d801/volumes" Oct 30 23:59:44.604551 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.604503 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522aa71b-47e4-44b1-825c-bf81305d21b4" path="/var/lib/kubelet/pods/522aa71b-47e4-44b1-825c-bf81305d21b4/volumes" Oct 30 23:59:44.605775 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.605752 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" event={"ID":"06e6c870-08ba-4b3c-a26d-45397d5ca730","Type":"ContainerDied","Data":"aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b"} Oct 30 23:59:44.605862 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.605784 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5" event={"ID":"06e6c870-08ba-4b3c-a26d-45397d5ca730","Type":"ContainerDied","Data":"a917798093e9458cf04e54e4797867c08c1f8671347231299a14dceb5da18525"} Oct 30 23:59:44.605862 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.605806 7570 scope.go:122] "RemoveContainer" containerID="aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b" Oct 30 23:59:44.621356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.621327 7570 scope.go:122] "RemoveContainer" containerID="5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51" Oct 30 23:59:44.625050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.624537 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5"] Oct 30 23:59:44.629587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.629559 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1bc837-8966d7f49-mf9r5"] Oct 30 23:59:44.635320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.635289 7570 scope.go:122] "RemoveContainer" containerID="aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b" Oct 30 23:59:44.635614 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:44.635593 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b\": container with ID starting with aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b not found: ID does not exist" containerID="aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b" Oct 30 23:59:44.635665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.635627 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b"} err="failed to get container status \"aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b\": rpc error: code = NotFound desc = could not find container \"aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b\": container with ID starting with aa5ae4033792abcce7a9f7f26581e7f8371b26b7991b200d3ed97a785e094f0b not found: ID does not exist" Oct 30 23:59:44.635665 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.635644 7570 scope.go:122] "RemoveContainer" containerID="5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51" Oct 30 23:59:44.635921 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:44.635901 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51\": container with ID starting with 5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51 not found: ID does not exist" containerID="5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51" Oct 30 23:59:44.635977 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:44.635928 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51"} err="failed to get container status \"5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51\": rpc error: code = NotFound desc = could not find container \"5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51\": container with ID starting with 5f34303b1f5c018fdfe195278034654869ae15616edf89e48f0e698e4ba17e51 not found: ID does not exist" Oct 30 23:59:46.327058 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.327016 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" podUID="d387eda1-c364-4443-ba0d-619368f57f28" containerName="payload-pre-processing" containerID="cri-o://23e7358568ccd8720464b8ec3cfa6637e23cbe8a66959e3052dd5bbddf4c7a4a" gracePeriod=25 Oct 30 23:59:46.604187 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.604162 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" path="/var/lib/kubelet/pods/06e6c870-08ba-4b3c-a26d-45397d5ca730/volumes" Oct 30 23:59:46.618040 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.618015 7570 generic.go:356] "Generic (PLEG): container finished" podID="d387eda1-c364-4443-ba0d-619368f57f28" containerID="23e7358568ccd8720464b8ec3cfa6637e23cbe8a66959e3052dd5bbddf4c7a4a" exitCode=0 Oct 30 23:59:46.618186 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.618083 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" event={"ID":"d387eda1-c364-4443-ba0d-619368f57f28","Type":"ContainerDied","Data":"23e7358568ccd8720464b8ec3cfa6637e23cbe8a66959e3052dd5bbddf4c7a4a"} Oct 30 23:59:46.618186 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.618124 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" event={"ID":"d387eda1-c364-4443-ba0d-619368f57f28","Type":"ContainerDied","Data":"5d606fe2faaed4cf389dca4762738730526be205c7b2f186efced4a8f0e0dbd3"} Oct 30 23:59:46.618186 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.618139 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d606fe2faaed4cf389dca4762738730526be205c7b2f186efced4a8f0e0dbd3" Oct 30 23:59:46.628715 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.628697 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:59:46.695728 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.695697 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d387eda1-c364-4443-ba0d-619368f57f28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d387eda1-c364-4443-ba0d-619368f57f28-plugins-config-volume\") pod \"d387eda1-c364-4443-ba0d-619368f57f28\" (UID: \"d387eda1-c364-4443-ba0d-619368f57f28\") " Oct 30 23:59:46.695859 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.695847 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d387eda1-c364-4443-ba0d-619368f57f28-kube-api-access-b85vb\" (UniqueName: \"kubernetes.io/projected/d387eda1-c364-4443-ba0d-619368f57f28-kube-api-access-b85vb\") pod \"d387eda1-c364-4443-ba0d-619368f57f28\" (UID: \"d387eda1-c364-4443-ba0d-619368f57f28\") " Oct 30 23:59:46.696091 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.696070 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d387eda1-c364-4443-ba0d-619368f57f28-plugins-config-volume" pod "d387eda1-c364-4443-ba0d-619368f57f28" (UID: "d387eda1-c364-4443-ba0d-619368f57f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:46.696321 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.696303 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d387eda1-c364-4443-ba0d-619368f57f28-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.698083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.698063 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d387eda1-c364-4443-ba0d-619368f57f28-kube-api-access-b85vb" pod "d387eda1-c364-4443-ba0d-619368f57f28" (UID: "d387eda1-c364-4443-ba0d-619368f57f28"). InnerVolumeSpecName "kube-api-access-b85vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:46.798140 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:46.798113 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b85vb\" (UniqueName: \"kubernetes.io/projected/d387eda1-c364-4443-ba0d-619368f57f28-kube-api-access-b85vb\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:47.626323 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:47.626291 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb" Oct 30 23:59:47.651805 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:47.651773 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb"] Oct 30 23:59:47.661022 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:47.660997 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a3e876-f64bdcd67-g9nqb"] Oct 30 23:59:47.734855 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:47.734821 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" podUID="37d7f8cf-f311-4b98-a6b7-25a043ac78ae" containerName="payload-processing" containerID="cri-o://a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7" gracePeriod=25 Oct 30 23:59:47.876416 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:47.876346 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw"] Oct 30 23:59:47.885083 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:47.885057 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32732d-h8rvw"] Oct 30 23:59:48.047347 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.047305 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:59:48.120306 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.120277 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-kube-api-access-5tchs\" (UniqueName: \"kubernetes.io/projected/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-kube-api-access-5tchs\") pod \"37d7f8cf-f311-4b98-a6b7-25a043ac78ae\" (UID: \"37d7f8cf-f311-4b98-a6b7-25a043ac78ae\") " Oct 30 23:59:48.120586 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.120571 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-plugins-config-volume\") pod \"37d7f8cf-f311-4b98-a6b7-25a043ac78ae\" (UID: \"37d7f8cf-f311-4b98-a6b7-25a043ac78ae\") " Oct 30 23:59:48.120967 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.120942 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-plugins-config-volume" pod "37d7f8cf-f311-4b98-a6b7-25a043ac78ae" (UID: "37d7f8cf-f311-4b98-a6b7-25a043ac78ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:48.121102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.121087 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:48.122659 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.122640 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-kube-api-access-5tchs" pod "37d7f8cf-f311-4b98-a6b7-25a043ac78ae" (UID: "37d7f8cf-f311-4b98-a6b7-25a043ac78ae"). InnerVolumeSpecName "kube-api-access-5tchs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:48.222908 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.222882 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tchs\" (UniqueName: \"kubernetes.io/projected/37d7f8cf-f311-4b98-a6b7-25a043ac78ae-kube-api-access-5tchs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:48.603219 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.603191 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506e0a2c-7d48-45e8-981c-d58c4265ff83" path="/var/lib/kubelet/pods/506e0a2c-7d48-45e8-981c-d58c4265ff83/volumes" Oct 30 23:59:48.603728 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.603711 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d387eda1-c364-4443-ba0d-619368f57f28" path="/var/lib/kubelet/pods/d387eda1-c364-4443-ba0d-619368f57f28/volumes" Oct 30 23:59:48.635842 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.635812 7570 generic.go:356] "Generic (PLEG): container finished" podID="37d7f8cf-f311-4b98-a6b7-25a043ac78ae" containerID="a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7" exitCode=0 Oct 30 23:59:48.636179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.635858 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" Oct 30 23:59:48.636179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.635898 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" event={"ID":"37d7f8cf-f311-4b98-a6b7-25a043ac78ae","Type":"ContainerDied","Data":"a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7"} Oct 30 23:59:48.636179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.635953 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg" event={"ID":"37d7f8cf-f311-4b98-a6b7-25a043ac78ae","Type":"ContainerDied","Data":"71d638e5a06a653cff071eabe2527b1dada7d93894db7e64b3cd387645c9f920"} Oct 30 23:59:48.636179 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.635979 7570 scope.go:122] "RemoveContainer" containerID="a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7" Oct 30 23:59:48.647020 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.647003 7570 scope.go:122] "RemoveContainer" containerID="a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7" Oct 30 23:59:48.647307 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:48.647281 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7\": container with ID starting with a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7 not found: ID does not exist" containerID="a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7" Oct 30 23:59:48.647365 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.647334 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7"} err="failed to get container status \"a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7\": rpc error: code = NotFound desc = could not find container \"a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7\": container with ID starting with a8ca5de7d14b34302874cba46c71bd67bb4062a89424822c0c7de69232b5f4c7 not found: ID does not exist" Oct 30 23:59:48.686466 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.686442 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg"] Oct 30 23:59:48.692971 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:48.692946 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32732d-6d57c5df4f-d55dg"] Oct 30 23:59:49.400012 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.399978 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq"] Oct 30 23:59:49.400695 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400681 7570 cpu_manager.go:414] "containerMap: removing container" podUID="37d7f8cf-f311-4b98-a6b7-25a043ac78ae" containerName="payload-processing" Oct 30 23:59:49.400734 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400697 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d7f8cf-f311-4b98-a6b7-25a043ac78ae" containerName="payload-processing" Oct 30 23:59:49.400734 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400713 7570 cpu_manager.go:414] "containerMap: removing container" podUID="d387eda1-c364-4443-ba0d-619368f57f28" containerName="payload-pre-processing" Oct 30 23:59:49.400734 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400718 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d387eda1-c364-4443-ba0d-619368f57f28" containerName="payload-pre-processing" Oct 30 23:59:49.400734 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400725 7570 cpu_manager.go:414] "containerMap: removing container" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" Oct 30 23:59:49.400734 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400730 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" Oct 30 23:59:49.400879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400741 7570 cpu_manager.go:414] "containerMap: removing container" podUID="522aa71b-47e4-44b1-825c-bf81305d21b4" containerName="payload-pre-processing" Oct 30 23:59:49.400879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400746 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522aa71b-47e4-44b1-825c-bf81305d21b4" containerName="payload-pre-processing" Oct 30 23:59:49.400879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400766 7570 cpu_manager.go:414] "containerMap: removing container" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" Oct 30 23:59:49.400879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400770 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" Oct 30 23:59:49.400879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400781 7570 cpu_manager.go:414] "containerMap: removing container" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" Oct 30 23:59:49.400879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400785 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" Oct 30 23:59:49.400879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400814 7570 cpu_manager.go:414] "containerMap: removing container" podUID="6530eea0-8758-41fb-8249-a151d46c8ebf" containerName="revoke-keys" Oct 30 23:59:49.400879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400818 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6530eea0-8758-41fb-8249-a151d46c8ebf" containerName="revoke-keys" Oct 30 23:59:49.400879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400824 7570 cpu_manager.go:414] "containerMap: removing container" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" Oct 30 23:59:49.400879 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400828 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" Oct 30 23:59:49.401120 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400920 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" Oct 30 23:59:49.401120 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400933 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6530eea0-8758-41fb-8249-a151d46c8ebf" containerName="revoke-keys" Oct 30 23:59:49.401120 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400939 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" Oct 30 23:59:49.401120 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400944 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d7f8cf-f311-4b98-a6b7-25a043ac78ae" containerName="payload-processing" Oct 30 23:59:49.401120 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400951 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="d387eda1-c364-4443-ba0d-619368f57f28" containerName="payload-pre-processing" Oct 30 23:59:49.401120 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400957 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e6c870-08ba-4b3c-a26d-45397d5ca730" containerName="maas-api" Oct 30 23:59:49.401120 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400964 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="522aa71b-47e4-44b1-825c-bf81305d21b4" containerName="payload-pre-processing" Oct 30 23:59:49.401120 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.400977 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3856a78-100e-48e0-aa1e-9a04bb10c99d" containerName="istio-proxy" Oct 30 23:59:49.406035 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.406018 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:49.414149 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.414124 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:59:49.414367 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.414254 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:59:49.420901 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.420873 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq"] Oct 30 23:59:49.548102 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.548072 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65510995-3f6f-46ad-a509-a253f117b668-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq\" (UID: \"65510995-3f6f-46ad-a509-a253f117b668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:49.548267 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.548114 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65510995-3f6f-46ad-a509-a253f117b668-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq\" (UID: \"65510995-3f6f-46ad-a509-a253f117b668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:49.548326 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.548302 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dppp6\" (UniqueName: \"kubernetes.io/projected/65510995-3f6f-46ad-a509-a253f117b668-kube-api-access-dppp6\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq\" (UID: \"65510995-3f6f-46ad-a509-a253f117b668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:49.649822 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.649797 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65510995-3f6f-46ad-a509-a253f117b668-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq\" (UID: \"65510995-3f6f-46ad-a509-a253f117b668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:49.650169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.649832 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65510995-3f6f-46ad-a509-a253f117b668-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq\" (UID: \"65510995-3f6f-46ad-a509-a253f117b668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:49.650169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.649883 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dppp6\" (UniqueName: \"kubernetes.io/projected/65510995-3f6f-46ad-a509-a253f117b668-kube-api-access-dppp6\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq\" (UID: \"65510995-3f6f-46ad-a509-a253f117b668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:49.650529 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.650510 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65510995-3f6f-46ad-a509-a253f117b668-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq\" (UID: \"65510995-3f6f-46ad-a509-a253f117b668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:49.652344 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.652325 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65510995-3f6f-46ad-a509-a253f117b668-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq\" (UID: \"65510995-3f6f-46ad-a509-a253f117b668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:49.666719 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.666699 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dppp6\" (UniqueName: \"kubernetes.io/projected/65510995-3f6f-46ad-a509-a253f117b668-kube-api-access-dppp6\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq\" (UID: \"65510995-3f6f-46ad-a509-a253f117b668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:49.724528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.724496 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:49.854363 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.854339 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:59:49.955141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.955067 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-kube-api-access-rhlvw\" (UniqueName: \"kubernetes.io/projected/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-kube-api-access-rhlvw\") pod \"7371f1ff-163e-49b2-a4ea-5a9bf3a18215\" (UID: \"7371f1ff-163e-49b2-a4ea-5a9bf3a18215\") " Oct 30 23:59:49.955374 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.955361 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-plugins-config-volume\") pod \"7371f1ff-163e-49b2-a4ea-5a9bf3a18215\" (UID: \"7371f1ff-163e-49b2-a4ea-5a9bf3a18215\") " Oct 30 23:59:49.955685 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.955659 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-plugins-config-volume" pod "7371f1ff-163e-49b2-a4ea-5a9bf3a18215" (UID: "7371f1ff-163e-49b2-a4ea-5a9bf3a18215"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:49.955969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.955925 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.957433 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:49.957410 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-kube-api-access-rhlvw" pod "7371f1ff-163e-49b2-a4ea-5a9bf3a18215" (UID: "7371f1ff-163e-49b2-a4ea-5a9bf3a18215"). InnerVolumeSpecName "kube-api-access-rhlvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:50.043439 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.043415 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq"] Oct 30 23:59:50.043621 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:59:50.043600 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65510995_3f6f_46ad_a509_a253f117b668.slice/crio-ef03d7fc7e6837d8245355e882e34b5dcfc93092927f4bf49792baeefddc3003 WatchSource:0}: Error finding container ef03d7fc7e6837d8245355e882e34b5dcfc93092927f4bf49792baeefddc3003: Status 404 returned error can't find the container with id ef03d7fc7e6837d8245355e882e34b5dcfc93092927f4bf49792baeefddc3003 Oct 30 23:59:50.057089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.057068 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhlvw\" (UniqueName: \"kubernetes.io/projected/7371f1ff-163e-49b2-a4ea-5a9bf3a18215-kube-api-access-rhlvw\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:50.604407 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.604373 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d7f8cf-f311-4b98-a6b7-25a043ac78ae" path="/var/lib/kubelet/pods/37d7f8cf-f311-4b98-a6b7-25a043ac78ae/volumes" Oct 30 23:59:50.654771 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.654744 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" event={"ID":"65510995-3f6f-46ad-a509-a253f117b668","Type":"ContainerStarted","Data":"bb710b18e03c8521ecbad0d2d43f7fd329c53203d225ed9b478bc4a747be60b6"} Oct 30 23:59:50.655188 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.654800 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" event={"ID":"65510995-3f6f-46ad-a509-a253f117b668","Type":"ContainerStarted","Data":"ef03d7fc7e6837d8245355e882e34b5dcfc93092927f4bf49792baeefddc3003"} Oct 30 23:59:50.656152 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.656129 7570 generic.go:356] "Generic (PLEG): container finished" podID="7371f1ff-163e-49b2-a4ea-5a9bf3a18215" containerID="e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c" exitCode=0 Oct 30 23:59:50.656240 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.656169 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" Oct 30 23:59:50.656240 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.656212 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" event={"ID":"7371f1ff-163e-49b2-a4ea-5a9bf3a18215","Type":"ContainerDied","Data":"e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c"} Oct 30 23:59:50.656300 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.656255 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt" event={"ID":"7371f1ff-163e-49b2-a4ea-5a9bf3a18215","Type":"ContainerDied","Data":"caef89725f86ed7c0ca0ee95652a0fcf1d96ebb1277189ff6746b7d1489cd461"} Oct 30 23:59:50.656300 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.656278 7570 scope.go:122] "RemoveContainer" containerID="e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c" Oct 30 23:59:50.671223 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.671205 7570 scope.go:122] "RemoveContainer" containerID="e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c" Oct 30 23:59:50.671582 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:50.671556 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c\": container with ID starting with e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c not found: ID does not exist" containerID="e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c" Oct 30 23:59:50.671657 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.671586 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c"} err="failed to get container status \"e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c\": rpc error: code = NotFound desc = could not find container \"e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c\": container with ID starting with e6e109d46b16f87554c3097883fcb801e5f09e675d7d209c1fc8aa68cf3ffd3c not found: ID does not exist" Oct 30 23:59:50.693443 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.693396 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" podStartSLOduration=1.693379981 podStartE2EDuration="1.693379981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:50.688011391 +0000 UTC m=+2402.207012478" watchObservedRunningTime="2026-10-30 23:59:50.693379981 +0000 UTC m=+2402.212381071" Oct 30 23:59:50.705577 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.705522 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt"] Oct 30 23:59:50.714048 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:50.714025 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-611ef3-6f8476d97f-2pftt"] Oct 30 23:59:51.672244 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:51.671912 7570 generic.go:356] "Generic (PLEG): container finished" podID="65510995-3f6f-46ad-a509-a253f117b668" containerID="bb710b18e03c8521ecbad0d2d43f7fd329c53203d225ed9b478bc4a747be60b6" exitCode=0 Oct 30 23:59:51.672869 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:51.672435 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" event={"ID":"65510995-3f6f-46ad-a509-a253f117b668","Type":"ContainerDied","Data":"bb710b18e03c8521ecbad0d2d43f7fd329c53203d225ed9b478bc4a747be60b6"} Oct 30 23:59:51.677254 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:51.677230 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5"] Oct 30 23:59:51.677524 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:51.677498 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" containerID="cri-o://941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad" gracePeriod=30 Oct 30 23:59:51.991994 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:51.991972 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:59:52.094485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.094118 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-metrics-certs\") pod \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " Oct 30 23:59:52.094485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.094263 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-maas-api-tls\") pod \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " Oct 30 23:59:52.094485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.094310 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-kube-api-access-zgqxt\" (UniqueName: \"kubernetes.io/projected/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-kube-api-access-zgqxt\") pod \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " Oct 30 23:59:52.094485 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.094406 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-openshift-service-ca\") pod \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\" (UID: \"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5\") " Oct 30 23:59:52.116141 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.109334 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-openshift-service-ca" pod "e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" (UID: "e8c5ef3a-fb2d-4f45-9d16-470f553b62c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:52.140676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.132863 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-maas-api-tls" pod "e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" (UID: "e8c5ef3a-fb2d-4f45-9d16-470f553b62c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:52.140676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.137073 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-metrics-certs" pod "e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" (UID: "e8c5ef3a-fb2d-4f45-9d16-470f553b62c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:52.140676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.139005 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-kube-api-access-zgqxt" pod "e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" (UID: "e8c5ef3a-fb2d-4f45-9d16-470f553b62c5"). InnerVolumeSpecName "kube-api-access-zgqxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:52.200988 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.200959 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:52.200988 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.200987 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:52.201229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.201001 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgqxt\" (UniqueName: \"kubernetes.io/projected/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-kube-api-access-zgqxt\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:52.201229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.201013 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:52.604089 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.604060 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7371f1ff-163e-49b2-a4ea-5a9bf3a18215" path="/var/lib/kubelet/pods/7371f1ff-163e-49b2-a4ea-5a9bf3a18215/volumes" Oct 30 23:59:52.690917 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.690874 7570 generic.go:356] "Generic (PLEG): container finished" podID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerID="941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad" exitCode=0 Oct 30 23:59:52.691504 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.690992 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" event={"ID":"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5","Type":"ContainerDied","Data":"941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad"} Oct 30 23:59:52.691642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.691069 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" Oct 30 23:59:52.691642 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.691623 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5" event={"ID":"e8c5ef3a-fb2d-4f45-9d16-470f553b62c5","Type":"ContainerDied","Data":"ad97c9bf05fe085156a16bd9de7d5c980f08d918cd4ac803e90e7aabb9266386"} Oct 30 23:59:52.691759 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.691686 7570 scope.go:122] "RemoveContainer" containerID="941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad" Oct 30 23:59:52.725683 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.725648 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5"] Oct 30 23:59:52.730184 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.730161 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a3e876-6c9bd4fb8b-dm9q5"] Oct 30 23:59:52.740169 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.740152 7570 scope.go:122] "RemoveContainer" containerID="8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4" Oct 30 23:59:52.752225 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.752203 7570 scope.go:122] "RemoveContainer" containerID="941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad" Oct 30 23:59:52.752709 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:52.752684 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad\": container with ID starting with 941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad not found: ID does not exist" containerID="941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad" Oct 30 23:59:52.752787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.752716 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad"} err="failed to get container status \"941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad\": rpc error: code = NotFound desc = could not find container \"941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad\": container with ID starting with 941e953a370ec0579bf7310f447c5d991f6443eab004addd0d0255ea07761dad not found: ID does not exist" Oct 30 23:59:52.752787 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.752735 7570 scope.go:122] "RemoveContainer" containerID="8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4" Oct 30 23:59:52.753010 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:52.752989 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4\": container with ID starting with 8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4 not found: ID does not exist" containerID="8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4" Oct 30 23:59:52.753077 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.753017 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4"} err="failed to get container status \"8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4\": rpc error: code = NotFound desc = could not find container \"8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4\": container with ID starting with 8f0721ab364f246bc22b592b7bed40370b2bb4bf7e5507d62d63c93fc2ac7cb4 not found: ID does not exist" Oct 30 23:59:52.855512 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:52.855431 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:53.012311 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.012282 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65510995-3f6f-46ad-a509-a253f117b668-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65510995-3f6f-46ad-a509-a253f117b668-maas-api-openshift-ca\") pod \"65510995-3f6f-46ad-a509-a253f117b668\" (UID: \"65510995-3f6f-46ad-a509-a253f117b668\") " Oct 30 23:59:53.012531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.012331 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65510995-3f6f-46ad-a509-a253f117b668-kube-api-access-dppp6\" (UniqueName: \"kubernetes.io/projected/65510995-3f6f-46ad-a509-a253f117b668-kube-api-access-dppp6\") pod \"65510995-3f6f-46ad-a509-a253f117b668\" (UID: \"65510995-3f6f-46ad-a509-a253f117b668\") " Oct 30 23:59:53.012531 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.012438 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65510995-3f6f-46ad-a509-a253f117b668-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65510995-3f6f-46ad-a509-a253f117b668-maas-api-xks-ca\") pod \"65510995-3f6f-46ad-a509-a253f117b668\" (UID: \"65510995-3f6f-46ad-a509-a253f117b668\") " Oct 30 23:59:53.012676 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.012650 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65510995-3f6f-46ad-a509-a253f117b668-maas-api-openshift-ca" pod "65510995-3f6f-46ad-a509-a253f117b668" (UID: "65510995-3f6f-46ad-a509-a253f117b668"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:53.013064 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.013043 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65510995-3f6f-46ad-a509-a253f117b668-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:53.014635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.014615 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65510995-3f6f-46ad-a509-a253f117b668-maas-api-xks-ca" pod "65510995-3f6f-46ad-a509-a253f117b668" (UID: "65510995-3f6f-46ad-a509-a253f117b668"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:53.014778 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.014760 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65510995-3f6f-46ad-a509-a253f117b668-kube-api-access-dppp6" pod "65510995-3f6f-46ad-a509-a253f117b668" (UID: "65510995-3f6f-46ad-a509-a253f117b668"). InnerVolumeSpecName "kube-api-access-dppp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:53.114528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.114441 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dppp6\" (UniqueName: \"kubernetes.io/projected/65510995-3f6f-46ad-a509-a253f117b668-kube-api-access-dppp6\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:53.114528 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.114512 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65510995-3f6f-46ad-a509-a253f117b668-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:53.414320 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.414226 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk"] Oct 30 23:59:53.414516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.414492 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" containerID="cri-o://94c4b43eba16793a73bbb2eec7436a8d641d6293bbd05ef1dc2359a826b91d3d" gracePeriod=30 Oct 30 23:59:53.701196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.701116 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" Oct 30 23:59:53.701196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.701125 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq" event={"ID":"65510995-3f6f-46ad-a509-a253f117b668","Type":"ContainerDied","Data":"ef03d7fc7e6837d8245355e882e34b5dcfc93092927f4bf49792baeefddc3003"} Oct 30 23:59:53.701196 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:53.701185 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef03d7fc7e6837d8245355e882e34b5dcfc93092927f4bf49792baeefddc3003" Oct 30 23:59:54.383540 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.383507 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4"] Oct 30 23:59:54.383791 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.383760 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" containerID="cri-o://73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d" gracePeriod=30 Oct 30 23:59:54.386167 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.386138 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28"] Oct 30 23:59:54.386377 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.386331 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" containerID="cri-o://359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79" gracePeriod=30 Oct 30 23:59:54.613880 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.613837 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" path="/var/lib/kubelet/pods/e8c5ef3a-fb2d-4f45-9d16-470f553b62c5/volumes" Oct 30 23:59:54.615170 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.615145 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl"] Oct 30 23:59:54.616530 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.616508 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" Oct 30 23:59:54.616646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.616632 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" Oct 30 23:59:54.616765 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.616754 7570 cpu_manager.go:414] "containerMap: removing container" podUID="65510995-3f6f-46ad-a509-a253f117b668" containerName="revoke-keys" Oct 30 23:59:54.616848 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.616837 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65510995-3f6f-46ad-a509-a253f117b668" containerName="revoke-keys" Oct 30 23:59:54.616998 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.616986 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" Oct 30 23:59:54.617076 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.617066 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" Oct 30 23:59:54.617161 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.617151 7570 cpu_manager.go:414] "containerMap: removing container" podUID="7371f1ff-163e-49b2-a4ea-5a9bf3a18215" containerName="payload-pre-processing" Oct 30 23:59:54.617226 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.617215 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7371f1ff-163e-49b2-a4ea-5a9bf3a18215" containerName="payload-pre-processing" Oct 30 23:59:54.617534 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.617517 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" Oct 30 23:59:54.617690 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.617676 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7371f1ff-163e-49b2-a4ea-5a9bf3a18215" containerName="payload-pre-processing" Oct 30 23:59:54.617796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.617785 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8c5ef3a-fb2d-4f45-9d16-470f553b62c5" containerName="maas-api" Oct 30 23:59:54.617890 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.617879 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="65510995-3f6f-46ad-a509-a253f117b668" containerName="revoke-keys" Oct 30 23:59:54.622932 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.622451 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:54.628705 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.626426 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl"] Oct 30 23:59:54.628705 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.627893 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:59:54.628705 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.628029 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:59:54.744687 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.744658 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl\" (UID: \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:54.745033 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.744720 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w424\" (UniqueName: \"kubernetes.io/projected/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-kube-api-access-8w424\") pod \"maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl\" (UID: \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:54.745033 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.744868 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl\" (UID: \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:54.846653 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.846612 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w424\" (UniqueName: \"kubernetes.io/projected/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-kube-api-access-8w424\") pod \"maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl\" (UID: \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:54.846829 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.846716 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl\" (UID: \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:54.846889 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.846866 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl\" (UID: \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:54.847537 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.847513 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl\" (UID: \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:54.849307 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.849288 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl\" (UID: \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:54.871902 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.871882 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w424\" (UniqueName: \"kubernetes.io/projected/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-kube-api-access-8w424\") pod \"maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl\" (UID: \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:54.946052 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:54.945963 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:55.244871 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:55.244843 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl"] Oct 30 23:59:55.245215 ip-10-0-137-81 kubenswrapper[7570]: W1030 23:59:55.245192 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f9c3b3_0d17_47f2_a0db_7aa3f1bc2ec4.slice/crio-421e7065fe4865ef4cac9d2f6d70d0e4418f0c656ffc24addd169e9887aee1df WatchSource:0}: Error finding container 421e7065fe4865ef4cac9d2f6d70d0e4418f0c656ffc24addd169e9887aee1df: Status 404 returned error can't find the container with id 421e7065fe4865ef4cac9d2f6d70d0e4418f0c656ffc24addd169e9887aee1df Oct 30 23:59:55.718906 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:55.718867 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" event={"ID":"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4","Type":"ContainerStarted","Data":"4906c75837cc7844108faedf93543b16de0b5b30f08d771d3188a7bbfb76b3c4"} Oct 30 23:59:55.719063 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:55.718934 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" event={"ID":"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4","Type":"ContainerStarted","Data":"421e7065fe4865ef4cac9d2f6d70d0e4418f0c656ffc24addd169e9887aee1df"} Oct 30 23:59:56.728063 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:56.728016 7570 generic.go:356] "Generic (PLEG): container finished" podID="56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4" containerID="4906c75837cc7844108faedf93543b16de0b5b30f08d771d3188a7bbfb76b3c4" exitCode=0 Oct 30 23:59:56.728483 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:56.728115 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" event={"ID":"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4","Type":"ContainerDied","Data":"4906c75837cc7844108faedf93543b16de0b5b30f08d771d3188a7bbfb76b3c4"} Oct 30 23:59:57.893525 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:57.893491 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:57.997325 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:57.997250 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-kube-api-access-8w424\" (UniqueName: \"kubernetes.io/projected/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-kube-api-access-8w424\") pod \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\" (UID: \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\") " Oct 30 23:59:57.997522 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:57.997328 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-openshift-ca\") pod \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\" (UID: \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\") " Oct 30 23:59:57.997576 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:57.997536 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-xks-ca\") pod \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\" (UID: \"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4\") " Oct 30 23:59:57.997737 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:57.997707 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-openshift-ca" pod "56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4" (UID: "56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:57.997990 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:57.997977 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:57.999633 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:57.999603 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-kube-api-access-8w424" pod "56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4" (UID: "56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4"). InnerVolumeSpecName "kube-api-access-8w424". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:57.999754 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:57.999736 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-xks-ca" pod "56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4" (UID: "56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:58.099613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.099586 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w424\" (UniqueName: \"kubernetes.io/projected/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-kube-api-access-8w424\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:58.099613 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.099612 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:58.152786 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.152763 7570 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:59:58.152865 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.152809 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:59:58.156726 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.156700 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:59:58.156796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.156754 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:59:58.172356 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.172331 7570 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:59:58.172442 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.172387 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:59:58.751989 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.751805 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" event={"ID":"56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4","Type":"ContainerDied","Data":"421e7065fe4865ef4cac9d2f6d70d0e4418f0c656ffc24addd169e9887aee1df"} Oct 30 23:59:58.751989 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.751876 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="421e7065fe4865ef4cac9d2f6d70d0e4418f0c656ffc24addd169e9887aee1df" Oct 30 23:59:58.751989 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.751970 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl" Oct 30 23:59:58.757915 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.757181 7570 generic.go:356] "Generic (PLEG): container finished" podID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerID="94c4b43eba16793a73bbb2eec7436a8d641d6293bbd05ef1dc2359a826b91d3d" exitCode=0 Oct 30 23:59:58.757915 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.757383 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" event={"ID":"51164b20-de0a-4b55-bdb7-7c25c2d5d922","Type":"ContainerDied","Data":"94c4b43eba16793a73bbb2eec7436a8d641d6293bbd05ef1dc2359a826b91d3d"} Oct 30 23:59:58.757915 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.757480 7570 scope.go:122] "RemoveContainer" containerID="982b2a7324033b2bc083f03b651ac5a85ec2d52e6b10437d961438545835dea8" Oct 30 23:59:58.766545 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.766522 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:59:58.793180 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.793149 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69"] Oct 30 23:59:58.793396 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.793372 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" containerID="cri-o://80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796" gracePeriod=30 Oct 30 23:59:58.810003 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.809980 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-data\") pod \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " Oct 30 23:59:58.810136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.810045 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-token\" (UniqueName: \"kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-token\") pod \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " Oct 30 23:59:58.810136 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.810077 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-kube-api-access-42jwx\" (UniqueName: \"kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-kube-api-access-42jwx\") pod \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " Oct 30 23:59:58.810229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.810142 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-certs\") pod \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " Oct 30 23:59:58.810229 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.810212 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istiod-ca-cert\") pod \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " Oct 30 23:59:58.810314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.810246 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-envoy\") pod \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " Oct 30 23:59:58.810314 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.810289 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-data" pod "51164b20-de0a-4b55-bdb7-7c25c2d5d922" (UID: "51164b20-de0a-4b55-bdb7-7c25c2d5d922"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:58.810395 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.810337 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-socket\") pod \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " Oct 30 23:59:58.810514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.810481 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-credential-socket\") pod \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " Oct 30 23:59:58.810514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.810510 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-podinfo\") pod \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\" (UID: \"51164b20-de0a-4b55-bdb7-7c25c2d5d922\") " Oct 30 23:59:58.810629 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.810574 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-certs" pod "51164b20-de0a-4b55-bdb7-7c25c2d5d922" (UID: "51164b20-de0a-4b55-bdb7-7c25c2d5d922"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:58.810993 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.810969 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istiod-ca-cert" pod "51164b20-de0a-4b55-bdb7-7c25c2d5d922" (UID: "51164b20-de0a-4b55-bdb7-7c25c2d5d922"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:58.811104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.811081 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-credential-socket" pod "51164b20-de0a-4b55-bdb7-7c25c2d5d922" (UID: "51164b20-de0a-4b55-bdb7-7c25c2d5d922"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:58.811505 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.811449 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:58.811606 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.811515 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:58.811606 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.811531 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:58.811606 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.811545 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:58.811786 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.811765 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-socket" pod "51164b20-de0a-4b55-bdb7-7c25c2d5d922" (UID: "51164b20-de0a-4b55-bdb7-7c25c2d5d922"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:58.813098 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.813074 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-token" pod "51164b20-de0a-4b55-bdb7-7c25c2d5d922" (UID: "51164b20-de0a-4b55-bdb7-7c25c2d5d922"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:58.813310 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.813293 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-envoy" pod "51164b20-de0a-4b55-bdb7-7c25c2d5d922" (UID: "51164b20-de0a-4b55-bdb7-7c25c2d5d922"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:58.813712 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.813691 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-podinfo" pod "51164b20-de0a-4b55-bdb7-7c25c2d5d922" (UID: "51164b20-de0a-4b55-bdb7-7c25c2d5d922"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:59:58.813899 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.813879 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-kube-api-access-42jwx" pod "51164b20-de0a-4b55-bdb7-7c25c2d5d922" (UID: "51164b20-de0a-4b55-bdb7-7c25c2d5d922"). InnerVolumeSpecName "kube-api-access-42jwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:58.915789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.915491 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:58.915789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.915531 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:58.915789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.915547 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:58.915789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.915561 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42jwx\" (UniqueName: \"kubernetes.io/projected/51164b20-de0a-4b55-bdb7-7c25c2d5d922-kube-api-access-42jwx\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:58.915789 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.915577 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51164b20-de0a-4b55-bdb7-7c25c2d5d922-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:58.997383 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.988130 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn"] Oct 30 23:59:58.999536 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:58.999242 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp"] Oct 30 23:59:59.008552 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.008442 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd"] Oct 30 23:59:59.416260 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:59.416221 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 30 23:59:59.419796 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.419771 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk"] Oct 30 23:59:59.420755 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.420737 7570 cpu_manager.go:414] "containerMap: removing container" podUID="56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4" containerName="revoke-keys" Oct 30 23:59:59.420820 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.420757 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4" containerName="revoke-keys" Oct 30 23:59:59.420820 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.420808 7570 cpu_manager.go:414] "containerMap: removing container" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" Oct 30 23:59:59.420820 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.420813 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" Oct 30 23:59:59.420924 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.420821 7570 cpu_manager.go:414] "containerMap: removing container" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" Oct 30 23:59:59.420924 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.420827 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" Oct 30 23:59:59.420983 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.420924 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" Oct 30 23:59:59.420983 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.420935 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" containerName="istio-proxy" Oct 30 23:59:59.420983 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.420945 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4" containerName="revoke-keys" Oct 30 23:59:59.424562 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.424545 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 30 23:59:59.431635 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.431615 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 30 23:59:59.431743 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.431704 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:59:59.436278 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.436253 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk"] Oct 30 23:59:59.531629 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.531602 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk\" (UID: \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 30 23:59:59.531725 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.531696 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59vcs\" (UniqueName: \"kubernetes.io/projected/42fd7f0b-dfc4-4e90-a321-c4a51115daab-kube-api-access-59vcs\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk\" (UID: \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 30 23:59:59.531864 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.531848 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk\" (UID: \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 30 23:59:59.633923 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.633886 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk\" (UID: \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 30 23:59:59.634159 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.634128 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk\" (UID: \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 30 23:59:59.634263 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.634188 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59vcs\" (UniqueName: \"kubernetes.io/projected/42fd7f0b-dfc4-4e90-a321-c4a51115daab-kube-api-access-59vcs\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk\" (UID: \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 30 23:59:59.638008 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.636527 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk\" (UID: \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 30 23:59:59.645514 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.639115 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk\" (UID: \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 30 23:59:59.654587 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.654553 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59vcs\" (UniqueName: \"kubernetes.io/projected/42fd7f0b-dfc4-4e90-a321-c4a51115daab-kube-api-access-59vcs\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk\" (UID: \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 30 23:59:59.690962 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.690940 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:59:59.705076 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.705055 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:59:59.736137 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.736103 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-podinfo\") pod \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " Oct 30 23:59:59.736305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.736154 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-certs\") pod \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " Oct 30 23:59:59.736305 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.736203 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-token\" (UniqueName: \"kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-token\") pod \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " Oct 30 23:59:59.736418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.736302 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-kube-api-access-nnbjz\" (UniqueName: \"kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-kube-api-access-nnbjz\") pod \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " Oct 30 23:59:59.736418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.736357 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-data\") pod \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " Oct 30 23:59:59.736418 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.736407 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-socket\") pod \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " Oct 30 23:59:59.736616 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.736579 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-credential-socket\") pod \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " Oct 30 23:59:59.736662 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.736627 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istiod-ca-cert\") pod \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " Oct 30 23:59:59.736815 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.736788 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-envoy\") pod \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\" (UID: \"e8fab750-eae1-4f10-b8f3-3db73b00a1f9\") " Oct 30 23:59:59.736970 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.736949 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-certs" pod "e8fab750-eae1-4f10-b8f3-3db73b00a1f9" (UID: "e8fab750-eae1-4f10-b8f3-3db73b00a1f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:59.737050 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.737024 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istiod-ca-cert" pod "e8fab750-eae1-4f10-b8f3-3db73b00a1f9" (UID: "e8fab750-eae1-4f10-b8f3-3db73b00a1f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:59.738886 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.737542 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.738886 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.737569 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.738886 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.737931 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-data" pod "e8fab750-eae1-4f10-b8f3-3db73b00a1f9" (UID: "e8fab750-eae1-4f10-b8f3-3db73b00a1f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:59.738886 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.738328 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-socket" pod "e8fab750-eae1-4f10-b8f3-3db73b00a1f9" (UID: "e8fab750-eae1-4f10-b8f3-3db73b00a1f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:59.738886 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.738599 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-credential-socket" pod "e8fab750-eae1-4f10-b8f3-3db73b00a1f9" (UID: "e8fab750-eae1-4f10-b8f3-3db73b00a1f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:59.740104 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.740002 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-token" pod "e8fab750-eae1-4f10-b8f3-3db73b00a1f9" (UID: "e8fab750-eae1-4f10-b8f3-3db73b00a1f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:59.740370 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.740334 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-envoy" pod "e8fab750-eae1-4f10-b8f3-3db73b00a1f9" (UID: "e8fab750-eae1-4f10-b8f3-3db73b00a1f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:59.742157 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.742135 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-kube-api-access-nnbjz" pod "e8fab750-eae1-4f10-b8f3-3db73b00a1f9" (UID: "e8fab750-eae1-4f10-b8f3-3db73b00a1f9"). InnerVolumeSpecName "kube-api-access-nnbjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:59.743092 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.743066 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-podinfo" pod "e8fab750-eae1-4f10-b8f3-3db73b00a1f9" (UID: "e8fab750-eae1-4f10-b8f3-3db73b00a1f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:59:59.788521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.786893 7570 generic.go:356] "Generic (PLEG): container finished" podID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerID="359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79" exitCode=0 Oct 30 23:59:59.788521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.786991 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" Oct 30 23:59:59.788521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.787002 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" event={"ID":"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb","Type":"ContainerDied","Data":"359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79"} Oct 30 23:59:59.788521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.787054 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28" event={"ID":"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb","Type":"ContainerDied","Data":"6b366f947309bc8d04548ad5f5d8fe848b3cb8ee38c9e999e7159962820e7305"} Oct 30 23:59:59.788521 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.787080 7570 scope.go:122] "RemoveContainer" containerID="359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79" Oct 30 23:59:59.789174 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.788999 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" event={"ID":"51164b20-de0a-4b55-bdb7-7c25c2d5d922","Type":"ContainerDied","Data":"73096d00f86e882a9154270a63f58b9ec11fd430d360328e89faa87b8232d85c"} Oct 30 23:59:59.789174 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.789120 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk" Oct 30 23:59:59.791310 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.791266 7570 generic.go:356] "Generic (PLEG): container finished" podID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerID="73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d" exitCode=0 Oct 30 23:59:59.791515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.791393 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" event={"ID":"e8fab750-eae1-4f10-b8f3-3db73b00a1f9","Type":"ContainerDied","Data":"73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d"} Oct 30 23:59:59.791515 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.791428 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" event={"ID":"e8fab750-eae1-4f10-b8f3-3db73b00a1f9","Type":"ContainerDied","Data":"f81d6eccbe6b875d8bfed0431580df91c7029cbb984e47ce4be672f8c2c2f1f6"} Oct 30 23:59:59.791660 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.791542 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4" Oct 30 23:59:59.812882 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.812862 7570 scope.go:122] "RemoveContainer" containerID="359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79" Oct 30 23:59:59.813254 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:59.813224 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79\": container with ID starting with 359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79 not found: ID does not exist" containerID="359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79" Oct 30 23:59:59.813402 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.813267 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79"} err="failed to get container status \"359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79\": rpc error: code = NotFound desc = could not find container \"359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79\": container with ID starting with 359f37026625e06aa1dbb8db0ef6740fb60b3543870b69c4aa0c0e83b88c6b79 not found: ID does not exist" Oct 30 23:59:59.813402 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.813291 7570 scope.go:122] "RemoveContainer" containerID="94c4b43eba16793a73bbb2eec7436a8d641d6293bbd05ef1dc2359a826b91d3d" Oct 30 23:59:59.818826 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.818808 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq"] Oct 30 23:59:59.822704 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.822686 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 30 23:59:59.826168 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.826151 7570 scope.go:122] "RemoveContainer" containerID="73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d" Oct 30 23:59:59.838081 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.838060 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-envoy\") pod \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " Oct 30 23:59:59.838156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.838108 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-data\") pod \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " Oct 30 23:59:59.838156 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.838137 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-kube-api-access-d474j\" (UniqueName: \"kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-kube-api-access-d474j\") pod \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " Oct 30 23:59:59.838281 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.838268 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-socket\") pod \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " Oct 30 23:59:59.838329 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.838314 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istiod-ca-cert\") pod \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " Oct 30 23:59:59.838403 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.838390 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-token\" (UniqueName: \"kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-token\") pod \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " Oct 30 23:59:59.838447 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.838418 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-podinfo\") pod \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " Oct 30 23:59:59.838516 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.838492 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-certs\") pod \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " Oct 30 23:59:59.838599 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.838580 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-data" pod "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" (UID: "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:59.838646 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.838615 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-credential-socket\") pod \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\" (UID: \"ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb\") " Oct 30 23:59:59.839096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.838917 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istiod-ca-cert" pod "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" (UID: "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:59.839096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.838986 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-socket" pod "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" (UID: "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:59.839096 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.839065 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-certs" pod "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" (UID: "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:59.839310 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.839218 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-credential-socket" pod "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" (UID: "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:59.839969 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.839940 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.840085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.839968 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.840085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.839987 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.840085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.840001 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.840085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.840015 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.840085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.840030 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.840085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.840044 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.840085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.840058 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.840085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.840072 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnbjz\" (UniqueName: \"kubernetes.io/projected/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-kube-api-access-nnbjz\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.840085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.840089 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.840085 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.840103 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.841112 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.840117 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fab750-eae1-4f10-b8f3-3db73b00a1f9-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.841274 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.841241 7570 scope.go:122] "RemoveContainer" containerID="dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca" Oct 30 23:59:59.843921 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.843222 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-envoy" pod "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" (UID: "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:59.844607 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.844248 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-kube-api-access-d474j" pod "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" (UID: "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb"). InnerVolumeSpecName "kube-api-access-d474j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:59.844607 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.844429 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-podinfo" pod "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" (UID: "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:59:59.844776 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.844750 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-token" pod "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" (UID: "ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:59.874703 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.874681 7570 scope.go:122] "RemoveContainer" containerID="73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d" Oct 30 23:59:59.875118 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:59.875086 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d\": container with ID starting with 73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d not found: ID does not exist" containerID="73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d" Oct 30 23:59:59.875221 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.875131 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d"} err="failed to get container status \"73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d\": rpc error: code = NotFound desc = could not find container \"73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d\": container with ID starting with 73ce30eb8b490caca0945609189eeab99b5df1687da1e83662209cce0552a12d not found: ID does not exist" Oct 30 23:59:59.875221 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.875153 7570 scope.go:122] "RemoveContainer" containerID="dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca" Oct 30 23:59:59.875616 ip-10-0-137-81 kubenswrapper[7570]: E1030 23:59:59.875590 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca\": container with ID starting with dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca not found: ID does not exist" containerID="dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca" Oct 30 23:59:59.875711 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.875625 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca"} err="failed to get container status \"dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca\": rpc error: code = NotFound desc = could not find container \"dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca\": container with ID starting with dc6ee30444e67333a251b324baed0dd8e5feff19c790c64d20a06b4a8560a7ca not found: ID does not exist" Oct 30 23:59:59.883878 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.883853 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk"] Oct 30 23:59:59.888798 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.888776 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-g4vbk"] Oct 30 23:59:59.923837 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.923803 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4"] Oct 30 23:59:59.941560 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.941483 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.941560 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.941519 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.941560 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.941547 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.941710 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.941563 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d474j\" (UniqueName: \"kubernetes.io/projected/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb-kube-api-access-d474j\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:59.943942 ip-10-0-137-81 kubenswrapper[7570]: I1030 23:59:59.943919 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-sq5q4"] Oct 31 00:00:00.174483 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.174440 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk"] Oct 31 00:00:00.179425 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:00:00.179396 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fd7f0b_dfc4_4e90_a321_c4a51115daab.slice/crio-82b94de27a52ae84ce5d50eba1f647104d2e12d9554566f316762ee046522d00 WatchSource:0}: Error finding container 82b94de27a52ae84ce5d50eba1f647104d2e12d9554566f316762ee046522d00: Status 404 returned error can't find the container with id 82b94de27a52ae84ce5d50eba1f647104d2e12d9554566f316762ee046522d00 Oct 31 00:00:00.197495 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.197436 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp"] Oct 31 00:00:00.198386 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.198367 7570 cpu_manager.go:414] "containerMap: removing container" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" Oct 31 00:00:00.198493 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.198390 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" Oct 31 00:00:00.198493 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.198433 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" Oct 31 00:00:00.198493 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.198441 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" Oct 31 00:00:00.198621 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.198608 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" containerName="istio-proxy" Oct 31 00:00:00.198652 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.198632 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" Oct 31 00:00:00.198652 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.198648 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" Oct 31 00:00:00.204139 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.204119 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" Oct 31 00:00:00.213528 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.213509 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch"] Oct 31 00:00:00.214432 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.214414 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" Oct 31 00:00:00.214500 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.214436 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" containerName="istio-proxy" Oct 31 00:00:00.218700 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.218683 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28"] Oct 31 00:00:00.218786 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.218774 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" Oct 31 00:00:00.224754 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.224735 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp"] Oct 31 00:00:00.240276 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.240254 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq"] Oct 31 00:00:00.253711 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.253685 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-tfg28"] Oct 31 00:00:00.254589 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.254570 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" Oct 31 00:00:00.254905 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.254880 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp\" (UID: \"c42b94b3-7dc2-465f-9e1a-f128b015e5b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" Oct 31 00:00:00.254984 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.254929 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz66p\" (UniqueName: \"kubernetes.io/projected/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-kube-api-access-xz66p\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp\" (UID: \"c42b94b3-7dc2-465f-9e1a-f128b015e5b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" Oct 31 00:00:00.261540 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.261445 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch"] Oct 31 00:00:00.394485 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.359973 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ffff16-0e62-47a6-9c29-4ad63ebbce79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq\" (UID: \"02ffff16-0e62-47a6-9c29-4ad63ebbce79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" Oct 31 00:00:00.394787 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.394755 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595cl\" (UniqueName: \"kubernetes.io/projected/02ffff16-0e62-47a6-9c29-4ad63ebbce79-kube-api-access-595cl\") pod \"maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq\" (UID: \"02ffff16-0e62-47a6-9c29-4ad63ebbce79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" Oct 31 00:00:00.395036 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.395020 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp\" (UID: \"c42b94b3-7dc2-465f-9e1a-f128b015e5b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" Oct 31 00:00:00.395142 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.395128 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz66p\" (UniqueName: \"kubernetes.io/projected/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-kube-api-access-xz66p\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp\" (UID: \"c42b94b3-7dc2-465f-9e1a-f128b015e5b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" Oct 31 00:00:00.398730 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.395267 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8gw\" (UniqueName: \"kubernetes.io/projected/e4a67598-601d-4304-999f-243f467eb6db-kube-api-access-ws8gw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch\" (UID: \"e4a67598-601d-4304-999f-243f467eb6db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" Oct 31 00:00:00.400666 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.400641 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a67598-601d-4304-999f-243f467eb6db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch\" (UID: \"e4a67598-601d-4304-999f-243f467eb6db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" Oct 31 00:00:00.400893 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.395966 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp\" (UID: \"c42b94b3-7dc2-465f-9e1a-f128b015e5b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" Oct 31 00:00:00.419455 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.419215 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq"] Oct 31 00:00:00.421613 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.421588 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29890080-w2p9r"] Oct 31 00:00:00.429487 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.426830 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-w2p9r" Oct 31 00:00:00.429487 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.427718 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z"] Oct 31 00:00:00.436521 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.434815 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh"] Oct 31 00:00:00.437149 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.437025 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" Oct 31 00:00:00.439835 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.439766 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Oct 31 00:00:00.439835 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.439812 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-ltwwh\"" Oct 31 00:00:00.450144 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.450122 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" Oct 31 00:00:00.452160 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.451785 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29890080-w2p9r"] Oct 31 00:00:00.469614 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.465365 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z"] Oct 31 00:00:00.478977 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.478801 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz66p\" (UniqueName: \"kubernetes.io/projected/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-kube-api-access-xz66p\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp\" (UID: \"c42b94b3-7dc2-465f-9e1a-f128b015e5b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" Oct 31 00:00:00.493574 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.493496 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh"] Oct 31 00:00:00.503106 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.501789 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8gw\" (UniqueName: \"kubernetes.io/projected/e4a67598-601d-4304-999f-243f467eb6db-kube-api-access-ws8gw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch\" (UID: \"e4a67598-601d-4304-999f-243f467eb6db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" Oct 31 00:00:00.503106 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.501909 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vrg\" (UniqueName: \"kubernetes.io/projected/de608395-f32c-4f76-ad97-988d2feb7ca0-kube-api-access-d4vrg\") pod \"image-pruner-29890080-w2p9r\" (UID: \"de608395-f32c-4f76-ad97-988d2feb7ca0\") " pod="openshift-image-registry/image-pruner-29890080-w2p9r" Oct 31 00:00:00.503106 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.501995 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a67598-601d-4304-999f-243f467eb6db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch\" (UID: \"e4a67598-601d-4304-999f-243f467eb6db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" Oct 31 00:00:00.503106 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.502031 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de608395-f32c-4f76-ad97-988d2feb7ca0-serviceca\") pod \"image-pruner-29890080-w2p9r\" (UID: \"de608395-f32c-4f76-ad97-988d2feb7ca0\") " pod="openshift-image-registry/image-pruner-29890080-w2p9r" Oct 31 00:00:00.503106 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.502053 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ffff16-0e62-47a6-9c29-4ad63ebbce79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq\" (UID: \"02ffff16-0e62-47a6-9c29-4ad63ebbce79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" Oct 31 00:00:00.503106 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.502074 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595cl\" (UniqueName: \"kubernetes.io/projected/02ffff16-0e62-47a6-9c29-4ad63ebbce79-kube-api-access-595cl\") pod \"maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq\" (UID: \"02ffff16-0e62-47a6-9c29-4ad63ebbce79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" Oct 31 00:00:00.503106 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.502140 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49g56\" (UniqueName: \"kubernetes.io/projected/1e93adf3-6136-459e-89cf-0006f8951204-kube-api-access-49g56\") pod \"maas-api-key-cleanup-29890080-mvk6z\" (UID: \"1e93adf3-6136-459e-89cf-0006f8951204\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" Oct 31 00:00:00.503106 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.502171 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e93adf3-6136-459e-89cf-0006f8951204-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890080-mvk6z\" (UID: \"1e93adf3-6136-459e-89cf-0006f8951204\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" Oct 31 00:00:00.504753 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.503881 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ffff16-0e62-47a6-9c29-4ad63ebbce79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq\" (UID: \"02ffff16-0e62-47a6-9c29-4ad63ebbce79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" Oct 31 00:00:00.522509 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.521120 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a67598-601d-4304-999f-243f467eb6db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch\" (UID: \"e4a67598-601d-4304-999f-243f467eb6db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" Oct 31 00:00:00.539225 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.538585 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8gw\" (UniqueName: \"kubernetes.io/projected/e4a67598-601d-4304-999f-243f467eb6db-kube-api-access-ws8gw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch\" (UID: \"e4a67598-601d-4304-999f-243f467eb6db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" Oct 31 00:00:00.552084 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.552055 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" Oct 31 00:00:00.561217 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.560924 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" Oct 31 00:00:00.568736 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.568128 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-595cl\" (UniqueName: \"kubernetes.io/projected/02ffff16-0e62-47a6-9c29-4ad63ebbce79-kube-api-access-595cl\") pod \"maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq\" (UID: \"02ffff16-0e62-47a6-9c29-4ad63ebbce79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" Oct 31 00:00:00.603822 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.603778 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de608395-f32c-4f76-ad97-988d2feb7ca0-serviceca\") pod \"image-pruner-29890080-w2p9r\" (UID: \"de608395-f32c-4f76-ad97-988d2feb7ca0\") " pod="openshift-image-registry/image-pruner-29890080-w2p9r" Oct 31 00:00:00.603973 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.603885 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b37651e-9dd9-4294-afa7-0bd3f427848c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh\" (UID: \"6b37651e-9dd9-4294-afa7-0bd3f427848c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" Oct 31 00:00:00.604219 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.604193 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49g56\" (UniqueName: \"kubernetes.io/projected/1e93adf3-6136-459e-89cf-0006f8951204-kube-api-access-49g56\") pod \"maas-api-key-cleanup-29890080-mvk6z\" (UID: \"1e93adf3-6136-459e-89cf-0006f8951204\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" Oct 31 00:00:00.604589 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.604563 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de608395-f32c-4f76-ad97-988d2feb7ca0-serviceca\") pod \"image-pruner-29890080-w2p9r\" (UID: \"de608395-f32c-4f76-ad97-988d2feb7ca0\") " pod="openshift-image-registry/image-pruner-29890080-w2p9r" Oct 31 00:00:00.605095 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.605070 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e93adf3-6136-459e-89cf-0006f8951204-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890080-mvk6z\" (UID: \"1e93adf3-6136-459e-89cf-0006f8951204\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" Oct 31 00:00:00.605241 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.605222 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxctl\" (UniqueName: \"kubernetes.io/projected/6b37651e-9dd9-4294-afa7-0bd3f427848c-kube-api-access-pxctl\") pod \"maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh\" (UID: \"6b37651e-9dd9-4294-afa7-0bd3f427848c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" Oct 31 00:00:00.605777 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.605756 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e93adf3-6136-459e-89cf-0006f8951204-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890080-mvk6z\" (UID: \"1e93adf3-6136-459e-89cf-0006f8951204\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" Oct 31 00:00:00.605930 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.605909 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vrg\" (UniqueName: \"kubernetes.io/projected/de608395-f32c-4f76-ad97-988d2feb7ca0-kube-api-access-d4vrg\") pod \"image-pruner-29890080-w2p9r\" (UID: \"de608395-f32c-4f76-ad97-988d2feb7ca0\") " pod="openshift-image-registry/image-pruner-29890080-w2p9r" Oct 31 00:00:00.608896 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.608868 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51164b20-de0a-4b55-bdb7-7c25c2d5d922" path="/var/lib/kubelet/pods/51164b20-de0a-4b55-bdb7-7c25c2d5d922/volumes" Oct 31 00:00:00.610777 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.610753 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb" path="/var/lib/kubelet/pods/ccbbfae7-a8ac-4546-b3e9-3867ce25cfeb/volumes" Oct 31 00:00:00.611562 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.611543 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8fab750-eae1-4f10-b8f3-3db73b00a1f9" path="/var/lib/kubelet/pods/e8fab750-eae1-4f10-b8f3-3db73b00a1f9/volumes" Oct 31 00:00:00.620604 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.620578 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49g56\" (UniqueName: \"kubernetes.io/projected/1e93adf3-6136-459e-89cf-0006f8951204-kube-api-access-49g56\") pod \"maas-api-key-cleanup-29890080-mvk6z\" (UID: \"1e93adf3-6136-459e-89cf-0006f8951204\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" Oct 31 00:00:00.622260 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.622239 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vrg\" (UniqueName: \"kubernetes.io/projected/de608395-f32c-4f76-ad97-988d2feb7ca0-kube-api-access-d4vrg\") pod \"image-pruner-29890080-w2p9r\" (UID: \"de608395-f32c-4f76-ad97-988d2feb7ca0\") " pod="openshift-image-registry/image-pruner-29890080-w2p9r" Oct 31 00:00:00.708070 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.708006 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxctl\" (UniqueName: \"kubernetes.io/projected/6b37651e-9dd9-4294-afa7-0bd3f427848c-kube-api-access-pxctl\") pod \"maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh\" (UID: \"6b37651e-9dd9-4294-afa7-0bd3f427848c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" Oct 31 00:00:00.708365 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.708341 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b37651e-9dd9-4294-afa7-0bd3f427848c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh\" (UID: \"6b37651e-9dd9-4294-afa7-0bd3f427848c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" Oct 31 00:00:00.709695 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.709677 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b37651e-9dd9-4294-afa7-0bd3f427848c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh\" (UID: \"6b37651e-9dd9-4294-afa7-0bd3f427848c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" Oct 31 00:00:00.725642 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.725452 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" Oct 31 00:00:00.728391 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.728366 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxctl\" (UniqueName: \"kubernetes.io/projected/6b37651e-9dd9-4294-afa7-0bd3f427848c-kube-api-access-pxctl\") pod \"maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh\" (UID: \"6b37651e-9dd9-4294-afa7-0bd3f427848c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" Oct 31 00:00:00.766109 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.765873 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-w2p9r" Oct 31 00:00:00.810349 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.810310 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" event={"ID":"42fd7f0b-dfc4-4e90-a321-c4a51115daab","Type":"ContainerStarted","Data":"6030f48318c3894c897c8c86533e531df88e1cc296d9190323ca6e8979d74118"} Oct 31 00:00:00.810515 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.810373 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" event={"ID":"42fd7f0b-dfc4-4e90-a321-c4a51115daab","Type":"ContainerStarted","Data":"82b94de27a52ae84ce5d50eba1f647104d2e12d9554566f316762ee046522d00"} Oct 31 00:00:00.855763 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.855530 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" Oct 31 00:00:00.861911 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.861884 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" Oct 31 00:00:00.922609 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.922583 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch"] Oct 31 00:00:00.930037 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.930002 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp"] Oct 31 00:00:00.936006 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:00.935955 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" podStartSLOduration=1.9359375810000001 podStartE2EDuration="1.935937581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:00.902163087 +0000 UTC m=+2412.421164180" watchObservedRunningTime="2026-10-31 00:00:00.935937581 +0000 UTC m=+2412.454938672" Oct 31 00:00:01.035712 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.035684 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq"] Oct 31 00:00:01.103933 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.103902 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29890080-w2p9r"] Oct 31 00:00:01.189769 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.189742 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z"] Oct 31 00:00:01.201599 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:00:01.201573 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde608395_f32c_4f76_ad97_988d2feb7ca0.slice/crio-75f02c333eeb428895392d95f6e07a038316dee50d0167efc7d41b05f3f870fb WatchSource:0}: Error finding container 75f02c333eeb428895392d95f6e07a038316dee50d0167efc7d41b05f3f870fb: Status 404 returned error can't find the container with id 75f02c333eeb428895392d95f6e07a038316dee50d0167efc7d41b05f3f870fb Oct 31 00:00:01.240302 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.240280 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh"] Oct 31 00:00:01.243937 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:00:01.243911 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b37651e_9dd9_4294_afa7_0bd3f427848c.slice/crio-e9ce7f94554e72d2a7fff9b4a66272ac575682b4acd69d644aa4cff4ec56d0f3 WatchSource:0}: Error finding container e9ce7f94554e72d2a7fff9b4a66272ac575682b4acd69d644aa4cff4ec56d0f3: Status 404 returned error can't find the container with id e9ce7f94554e72d2a7fff9b4a66272ac575682b4acd69d644aa4cff4ec56d0f3 Oct 31 00:00:01.819668 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.819642 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" event={"ID":"c42b94b3-7dc2-465f-9e1a-f128b015e5b8","Type":"ContainerStarted","Data":"bb33632cbf22df96b9eb2ea0f996bc655b953dcc86052951db304968e79770ee"} Oct 31 00:00:01.819795 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.819703 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" event={"ID":"c42b94b3-7dc2-465f-9e1a-f128b015e5b8","Type":"ContainerStarted","Data":"61f96186595630bf4226fdfeff75b45632492d8bab07458d843a2c714fa431b0"} Oct 31 00:00:01.821522 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.821494 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" event={"ID":"02ffff16-0e62-47a6-9c29-4ad63ebbce79","Type":"ContainerStarted","Data":"3d96b9481e016a78e6227e2d1aa0a0cf3e98a03d8ed988b9a2fe2ae75affeae2"} Oct 31 00:00:01.821630 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.821548 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" event={"ID":"02ffff16-0e62-47a6-9c29-4ad63ebbce79","Type":"ContainerStarted","Data":"bea4101a792241e049699e6fd76c747b13f9dbbe19ea99f9b870097e55c2bc2e"} Oct 31 00:00:01.823157 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.823135 7570 generic.go:356] "Generic (PLEG): container finished" podID="42fd7f0b-dfc4-4e90-a321-c4a51115daab" containerID="6030f48318c3894c897c8c86533e531df88e1cc296d9190323ca6e8979d74118" exitCode=0 Oct 31 00:00:01.823259 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.823200 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" event={"ID":"42fd7f0b-dfc4-4e90-a321-c4a51115daab","Type":"ContainerDied","Data":"6030f48318c3894c897c8c86533e531df88e1cc296d9190323ca6e8979d74118"} Oct 31 00:00:01.825210 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.825186 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" event={"ID":"1e93adf3-6136-459e-89cf-0006f8951204","Type":"ContainerStarted","Data":"f639954ade36bf5010db5dcf04400b4bc7f81de7d235ed2930a1f957a0f2518e"} Oct 31 00:00:01.825309 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.825233 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" event={"ID":"1e93adf3-6136-459e-89cf-0006f8951204","Type":"ContainerStarted","Data":"12f51ca8cb7bb83eb6ce14fafd81d25441b9bac047e9ae8e2055864a23e6df6c"} Oct 31 00:00:01.826749 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.826724 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" event={"ID":"e4a67598-601d-4304-999f-243f467eb6db","Type":"ContainerStarted","Data":"ad990ea5e755cc9acace5f23ff10441f15e0660023ace9dabd050fe8d9188345"} Oct 31 00:00:01.826837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.826767 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" event={"ID":"e4a67598-601d-4304-999f-243f467eb6db","Type":"ContainerStarted","Data":"597f9289adb57211c44a9a4cb3a6649ec3a1ee824593990baf7fe32046d2a918"} Oct 31 00:00:01.828455 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.828421 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-w2p9r" event={"ID":"de608395-f32c-4f76-ad97-988d2feb7ca0","Type":"ContainerStarted","Data":"a3bd631c08a5909a13ce5c0e927a275fdc10713a0f0948052e62516d0d8b2853"} Oct 31 00:00:01.828577 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.828508 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-w2p9r" event={"ID":"de608395-f32c-4f76-ad97-988d2feb7ca0","Type":"ContainerStarted","Data":"75f02c333eeb428895392d95f6e07a038316dee50d0167efc7d41b05f3f870fb"} Oct 31 00:00:01.829835 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.829817 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" event={"ID":"6b37651e-9dd9-4294-afa7-0bd3f427848c","Type":"ContainerStarted","Data":"ea0abdcf852c98a67fc3bb685675426144f67f1377932d884ece941b5315ad3a"} Oct 31 00:00:01.830040 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.829856 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" event={"ID":"6b37651e-9dd9-4294-afa7-0bd3f427848c","Type":"ContainerStarted","Data":"e9ce7f94554e72d2a7fff9b4a66272ac575682b4acd69d644aa4cff4ec56d0f3"} Oct 31 00:00:01.881908 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.881855 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" podStartSLOduration=1.881837756 podStartE2EDuration="1.881837756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:01.850204319 +0000 UTC m=+2413.369205408" watchObservedRunningTime="2026-10-31 00:00:01.881837756 +0000 UTC m=+2413.400838847" Oct 31 00:00:01.947950 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.947887 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29890080-w2p9r" podStartSLOduration=1.947872928 podStartE2EDuration="1.947872928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:01.91267402 +0000 UTC m=+2413.431675111" watchObservedRunningTime="2026-10-31 00:00:01.947872928 +0000 UTC m=+2413.466874205" Oct 31 00:00:01.950608 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.950564 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" podStartSLOduration=1.950551441 podStartE2EDuration="1.950551441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:01.946831742 +0000 UTC m=+2413.465832830" watchObservedRunningTime="2026-10-31 00:00:01.950551441 +0000 UTC m=+2413.469552530" Oct 31 00:00:01.974134 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:01.974084 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" podStartSLOduration=1.97406824 podStartE2EDuration="1.97406824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:01.97374153 +0000 UTC m=+2413.492742613" watchObservedRunningTime="2026-10-31 00:00:01.97406824 +0000 UTC m=+2413.493069331" Oct 31 00:00:02.002277 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.002227 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" podStartSLOduration=2.002209292 podStartE2EDuration="2.002209292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:01.999506876 +0000 UTC m=+2413.518507966" watchObservedRunningTime="2026-10-31 00:00:02.002209292 +0000 UTC m=+2413.521210382" Oct 31 00:00:02.839347 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.839292 7570 generic.go:356] "Generic (PLEG): container finished" podID="1e93adf3-6136-459e-89cf-0006f8951204" containerID="f639954ade36bf5010db5dcf04400b4bc7f81de7d235ed2930a1f957a0f2518e" exitCode=60 Oct 31 00:00:02.839606 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.839381 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" event={"ID":"1e93adf3-6136-459e-89cf-0006f8951204","Type":"ContainerDied","Data":"f639954ade36bf5010db5dcf04400b4bc7f81de7d235ed2930a1f957a0f2518e"} Oct 31 00:00:02.839813 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.839795 7570 scope.go:122] "RemoveContainer" containerID="f639954ade36bf5010db5dcf04400b4bc7f81de7d235ed2930a1f957a0f2518e" Oct 31 00:00:02.841366 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.841344 7570 generic.go:356] "Generic (PLEG): container finished" podID="e4a67598-601d-4304-999f-243f467eb6db" containerID="ad990ea5e755cc9acace5f23ff10441f15e0660023ace9dabd050fe8d9188345" exitCode=60 Oct 31 00:00:02.841515 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.841498 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" event={"ID":"e4a67598-601d-4304-999f-243f467eb6db","Type":"ContainerDied","Data":"ad990ea5e755cc9acace5f23ff10441f15e0660023ace9dabd050fe8d9188345"} Oct 31 00:00:02.841791 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.841774 7570 scope.go:122] "RemoveContainer" containerID="ad990ea5e755cc9acace5f23ff10441f15e0660023ace9dabd050fe8d9188345" Oct 31 00:00:02.843271 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.843238 7570 generic.go:356] "Generic (PLEG): container finished" podID="de608395-f32c-4f76-ad97-988d2feb7ca0" containerID="a3bd631c08a5909a13ce5c0e927a275fdc10713a0f0948052e62516d0d8b2853" exitCode=0 Oct 31 00:00:02.843371 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.843305 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-w2p9r" event={"ID":"de608395-f32c-4f76-ad97-988d2feb7ca0","Type":"ContainerDied","Data":"a3bd631c08a5909a13ce5c0e927a275fdc10713a0f0948052e62516d0d8b2853"} Oct 31 00:00:02.856431 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.856398 7570 generic.go:356] "Generic (PLEG): container finished" podID="6b37651e-9dd9-4294-afa7-0bd3f427848c" containerID="ea0abdcf852c98a67fc3bb685675426144f67f1377932d884ece941b5315ad3a" exitCode=60 Oct 31 00:00:02.856431 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.856450 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" event={"ID":"6b37651e-9dd9-4294-afa7-0bd3f427848c","Type":"ContainerDied","Data":"ea0abdcf852c98a67fc3bb685675426144f67f1377932d884ece941b5315ad3a"} Oct 31 00:00:02.856737 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.856723 7570 scope.go:122] "RemoveContainer" containerID="ea0abdcf852c98a67fc3bb685675426144f67f1377932d884ece941b5315ad3a" Oct 31 00:00:02.858221 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.858201 7570 generic.go:356] "Generic (PLEG): container finished" podID="c42b94b3-7dc2-465f-9e1a-f128b015e5b8" containerID="bb33632cbf22df96b9eb2ea0f996bc655b953dcc86052951db304968e79770ee" exitCode=60 Oct 31 00:00:02.858312 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.858289 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" event={"ID":"c42b94b3-7dc2-465f-9e1a-f128b015e5b8","Type":"ContainerDied","Data":"bb33632cbf22df96b9eb2ea0f996bc655b953dcc86052951db304968e79770ee"} Oct 31 00:00:02.858571 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.858550 7570 scope.go:122] "RemoveContainer" containerID="bb33632cbf22df96b9eb2ea0f996bc655b953dcc86052951db304968e79770ee" Oct 31 00:00:02.861079 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.861061 7570 generic.go:356] "Generic (PLEG): container finished" podID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" containerID="3d96b9481e016a78e6227e2d1aa0a0cf3e98a03d8ed988b9a2fe2ae75affeae2" exitCode=60 Oct 31 00:00:02.861159 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.861139 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" event={"ID":"02ffff16-0e62-47a6-9c29-4ad63ebbce79","Type":"ContainerDied","Data":"3d96b9481e016a78e6227e2d1aa0a0cf3e98a03d8ed988b9a2fe2ae75affeae2"} Oct 31 00:00:02.861338 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.861322 7570 scope.go:122] "RemoveContainer" containerID="3d96b9481e016a78e6227e2d1aa0a0cf3e98a03d8ed988b9a2fe2ae75affeae2" Oct 31 00:00:02.870933 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:02.870899 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" podStartSLOduration=2.8708869 podStartE2EDuration="2.8708869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:02.032701329 +0000 UTC m=+2413.551702419" watchObservedRunningTime="2026-10-31 00:00:02.8708869 +0000 UTC m=+2414.389887988" Oct 31 00:00:03.119104 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.119072 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 31 00:00:03.264337 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.264311 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fd7f0b-dfc4-4e90-a321-c4a51115daab-kube-api-access-59vcs\" (UniqueName: \"kubernetes.io/projected/42fd7f0b-dfc4-4e90-a321-c4a51115daab-kube-api-access-59vcs\") pod \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\" (UID: \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\") " Oct 31 00:00:03.264502 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.264377 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-openshift-ca\") pod \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\" (UID: \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\") " Oct 31 00:00:03.264502 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.264431 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-xks-ca\") pod \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\" (UID: \"42fd7f0b-dfc4-4e90-a321-c4a51115daab\") " Oct 31 00:00:03.264833 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.264797 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-openshift-ca" pod "42fd7f0b-dfc4-4e90-a321-c4a51115daab" (UID: "42fd7f0b-dfc4-4e90-a321-c4a51115daab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:03.265080 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.265065 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:03.266655 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.266632 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fd7f0b-dfc4-4e90-a321-c4a51115daab-kube-api-access-59vcs" pod "42fd7f0b-dfc4-4e90-a321-c4a51115daab" (UID: "42fd7f0b-dfc4-4e90-a321-c4a51115daab"). InnerVolumeSpecName "kube-api-access-59vcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:03.266750 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.266722 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-xks-ca" pod "42fd7f0b-dfc4-4e90-a321-c4a51115daab" (UID: "42fd7f0b-dfc4-4e90-a321-c4a51115daab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:03.366886 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.366829 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59vcs\" (UniqueName: \"kubernetes.io/projected/42fd7f0b-dfc4-4e90-a321-c4a51115daab-kube-api-access-59vcs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:03.366886 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.366853 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fd7f0b-dfc4-4e90-a321-c4a51115daab-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:03.873093 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.873054 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" event={"ID":"e4a67598-601d-4304-999f-243f467eb6db","Type":"ContainerStarted","Data":"b170d3de492a86cd130c500f3cf7098ccce143f309ee7c6e90f89dac6780bb80"} Oct 31 00:00:03.876210 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.876179 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" event={"ID":"6b37651e-9dd9-4294-afa7-0bd3f427848c","Type":"ContainerStarted","Data":"24dd4fe118cdc74eafaec2e9b78ea0e99bd4ab9ac38bf1ae4988b35178106d5a"} Oct 31 00:00:03.878695 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.878671 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" event={"ID":"c42b94b3-7dc2-465f-9e1a-f128b015e5b8","Type":"ContainerStarted","Data":"fda1213b39a77702c54d081a4092b797b99e6abbceec0c1cb3dbf96595428705"} Oct 31 00:00:03.880828 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.880809 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" event={"ID":"02ffff16-0e62-47a6-9c29-4ad63ebbce79","Type":"ContainerStarted","Data":"921986a36f6412b4724e6ee00833ab74fbf63002f2ac3a8b3774b330fafb51f9"} Oct 31 00:00:03.883207 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.883157 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" event={"ID":"42fd7f0b-dfc4-4e90-a321-c4a51115daab","Type":"ContainerDied","Data":"82b94de27a52ae84ce5d50eba1f647104d2e12d9554566f316762ee046522d00"} Oct 31 00:00:03.883420 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.883362 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82b94de27a52ae84ce5d50eba1f647104d2e12d9554566f316762ee046522d00" Oct 31 00:00:03.884885 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.883747 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk" Oct 31 00:00:03.888663 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:03.888624 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" event={"ID":"1e93adf3-6136-459e-89cf-0006f8951204","Type":"ContainerStarted","Data":"fd5890661ee62ac9462d31a77215b6f2ca7ff5c708beca4cb663aaaec083dc74"} Oct 31 00:00:04.160889 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.160862 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-w2p9r" Oct 31 00:00:04.169810 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.169784 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 31 00:00:04.178787 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.178763 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-socket\") pod \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " Oct 31 00:00:04.178905 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.178890 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-certs\") pod \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " Oct 31 00:00:04.178968 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.178923 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-envoy\") pod \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " Oct 31 00:00:04.179015 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.178990 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istiod-ca-cert\") pod \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " Oct 31 00:00:04.179015 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.178999 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-socket" pod "84a7842f-6043-4b93-ab32-d25e2ba8c5bb" (UID: "84a7842f-6043-4b93-ab32-d25e2ba8c5bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:04.179103 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.179074 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-credential-socket\") pod \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " Oct 31 00:00:04.179156 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.179138 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-certs" pod "84a7842f-6043-4b93-ab32-d25e2ba8c5bb" (UID: "84a7842f-6043-4b93-ab32-d25e2ba8c5bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:04.179210 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.179174 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-kube-api-access-qnsm4\" (UniqueName: \"kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-kube-api-access-qnsm4\") pod \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " Oct 31 00:00:04.179210 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.179197 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de608395-f32c-4f76-ad97-988d2feb7ca0-kube-api-access-d4vrg\" (UniqueName: \"kubernetes.io/projected/de608395-f32c-4f76-ad97-988d2feb7ca0-kube-api-access-d4vrg\") pod \"de608395-f32c-4f76-ad97-988d2feb7ca0\" (UID: \"de608395-f32c-4f76-ad97-988d2feb7ca0\") " Oct 31 00:00:04.179310 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.179251 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de608395-f32c-4f76-ad97-988d2feb7ca0-serviceca\" (UniqueName: \"kubernetes.io/configmap/de608395-f32c-4f76-ad97-988d2feb7ca0-serviceca\") pod \"de608395-f32c-4f76-ad97-988d2feb7ca0\" (UID: \"de608395-f32c-4f76-ad97-988d2feb7ca0\") " Oct 31 00:00:04.179310 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.179270 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-data\") pod \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " Oct 31 00:00:04.179310 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.179305 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-podinfo\") pod \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " Oct 31 00:00:04.179446 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.179373 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-credential-socket" pod "84a7842f-6043-4b93-ab32-d25e2ba8c5bb" (UID: "84a7842f-6043-4b93-ab32-d25e2ba8c5bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:04.179446 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.179391 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istiod-ca-cert" pod "84a7842f-6043-4b93-ab32-d25e2ba8c5bb" (UID: "84a7842f-6043-4b93-ab32-d25e2ba8c5bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:04.179446 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.179416 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-token\" (UniqueName: \"kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-token\") pod \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\" (UID: \"84a7842f-6043-4b93-ab32-d25e2ba8c5bb\") " Oct 31 00:00:04.179623 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.179555 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-data" pod "84a7842f-6043-4b93-ab32-d25e2ba8c5bb" (UID: "84a7842f-6043-4b93-ab32-d25e2ba8c5bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:04.179929 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.179906 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de608395-f32c-4f76-ad97-988d2feb7ca0-serviceca" pod "de608395-f32c-4f76-ad97-988d2feb7ca0" (UID: "de608395-f32c-4f76-ad97-988d2feb7ca0"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:04.180148 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.180089 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:04.180148 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.180114 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:04.180148 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.180128 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:04.180148 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.180141 7570 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de608395-f32c-4f76-ad97-988d2feb7ca0-serviceca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:04.180148 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.180150 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:04.180148 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.180158 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:04.182107 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.182084 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-kube-api-access-qnsm4" pod "84a7842f-6043-4b93-ab32-d25e2ba8c5bb" (UID: "84a7842f-6043-4b93-ab32-d25e2ba8c5bb"). InnerVolumeSpecName "kube-api-access-qnsm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:04.182621 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.182587 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-envoy" pod "84a7842f-6043-4b93-ab32-d25e2ba8c5bb" (UID: "84a7842f-6043-4b93-ab32-d25e2ba8c5bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:04.182621 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.182600 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-token" pod "84a7842f-6043-4b93-ab32-d25e2ba8c5bb" (UID: "84a7842f-6043-4b93-ab32-d25e2ba8c5bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:04.182805 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.182790 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de608395-f32c-4f76-ad97-988d2feb7ca0-kube-api-access-d4vrg" pod "de608395-f32c-4f76-ad97-988d2feb7ca0" (UID: "de608395-f32c-4f76-ad97-988d2feb7ca0"). InnerVolumeSpecName "kube-api-access-d4vrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:04.182896 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.182883 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-podinfo" pod "84a7842f-6043-4b93-ab32-d25e2ba8c5bb" (UID: "84a7842f-6043-4b93-ab32-d25e2ba8c5bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:00:04.282076 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.282050 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:04.282076 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.282079 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:04.282217 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.282090 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnsm4\" (UniqueName: \"kubernetes.io/projected/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-kube-api-access-qnsm4\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:04.282217 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.282100 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4vrg\" (UniqueName: \"kubernetes.io/projected/de608395-f32c-4f76-ad97-988d2feb7ca0-kube-api-access-d4vrg\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:04.282217 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.282109 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a7842f-6043-4b93-ab32-d25e2ba8c5bb-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:04.384053 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.383955 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq"] Oct 31 00:00:04.384189 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.384162 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" containerID="cri-o://5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357" gracePeriod=30 Oct 31 00:00:04.819579 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.819494 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" podUID="201fbcb8-8fed-44e3-b60d-4b89d8c114ca" containerName="payload-processing" containerID="cri-o://a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f" gracePeriod=25 Oct 31 00:00:04.898391 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.898370 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-w2p9r" Oct 31 00:00:04.898541 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.898395 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-w2p9r" event={"ID":"de608395-f32c-4f76-ad97-988d2feb7ca0","Type":"ContainerDied","Data":"75f02c333eeb428895392d95f6e07a038316dee50d0167efc7d41b05f3f870fb"} Oct 31 00:00:04.898541 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.898448 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75f02c333eeb428895392d95f6e07a038316dee50d0167efc7d41b05f3f870fb" Oct 31 00:00:04.900755 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.900735 7570 generic.go:356] "Generic (PLEG): container finished" podID="6b37651e-9dd9-4294-afa7-0bd3f427848c" containerID="24dd4fe118cdc74eafaec2e9b78ea0e99bd4ab9ac38bf1ae4988b35178106d5a" exitCode=60 Oct 31 00:00:04.900850 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.900820 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" event={"ID":"6b37651e-9dd9-4294-afa7-0bd3f427848c","Type":"ContainerDied","Data":"24dd4fe118cdc74eafaec2e9b78ea0e99bd4ab9ac38bf1ae4988b35178106d5a"} Oct 31 00:00:04.900916 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.900871 7570 scope.go:122] "RemoveContainer" containerID="ea0abdcf852c98a67fc3bb685675426144f67f1377932d884ece941b5315ad3a" Oct 31 00:00:04.901076 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.901055 7570 scope.go:122] "RemoveContainer" containerID="24dd4fe118cdc74eafaec2e9b78ea0e99bd4ab9ac38bf1ae4988b35178106d5a" Oct 31 00:00:04.901683 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:04.901358 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh_odh-ai-gateway-infra(6b37651e-9dd9-4294-afa7-0bd3f427848c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" Oct 31 00:00:04.903095 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.903078 7570 generic.go:356] "Generic (PLEG): container finished" podID="c42b94b3-7dc2-465f-9e1a-f128b015e5b8" containerID="fda1213b39a77702c54d081a4092b797b99e6abbceec0c1cb3dbf96595428705" exitCode=60 Oct 31 00:00:04.903169 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.903143 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" event={"ID":"c42b94b3-7dc2-465f-9e1a-f128b015e5b8","Type":"ContainerDied","Data":"fda1213b39a77702c54d081a4092b797b99e6abbceec0c1cb3dbf96595428705"} Oct 31 00:00:04.903364 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.903351 7570 scope.go:122] "RemoveContainer" containerID="fda1213b39a77702c54d081a4092b797b99e6abbceec0c1cb3dbf96595428705" Oct 31 00:00:04.903617 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:04.903593 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp_odh-ai-gateway-infra(c42b94b3-7dc2-465f-9e1a-f128b015e5b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" podUID="c42b94b3-7dc2-465f-9e1a-f128b015e5b8" Oct 31 00:00:04.905034 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.905012 7570 generic.go:356] "Generic (PLEG): container finished" podID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" containerID="921986a36f6412b4724e6ee00833ab74fbf63002f2ac3a8b3774b330fafb51f9" exitCode=60 Oct 31 00:00:04.905100 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.905061 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" event={"ID":"02ffff16-0e62-47a6-9c29-4ad63ebbce79","Type":"ContainerDied","Data":"921986a36f6412b4724e6ee00833ab74fbf63002f2ac3a8b3774b330fafb51f9"} Oct 31 00:00:04.905289 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.905277 7570 scope.go:122] "RemoveContainer" containerID="921986a36f6412b4724e6ee00833ab74fbf63002f2ac3a8b3774b330fafb51f9" Oct 31 00:00:04.905526 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:04.905509 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq_odh-ai-gateway-infra(02ffff16-0e62-47a6-9c29-4ad63ebbce79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" Oct 31 00:00:04.906868 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.906851 7570 generic.go:356] "Generic (PLEG): container finished" podID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerID="80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796" exitCode=0 Oct 31 00:00:04.906945 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.906902 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" Oct 31 00:00:04.906945 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.906922 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" event={"ID":"84a7842f-6043-4b93-ab32-d25e2ba8c5bb","Type":"ContainerDied","Data":"80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796"} Oct 31 00:00:04.907018 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.906973 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69" event={"ID":"84a7842f-6043-4b93-ab32-d25e2ba8c5bb","Type":"ContainerDied","Data":"10e5e2a6307b59b62b3433b9f7fcd74596640f4cfa642f931e0d87d3f4749f90"} Oct 31 00:00:04.909126 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.909108 7570 generic.go:356] "Generic (PLEG): container finished" podID="1e93adf3-6136-459e-89cf-0006f8951204" containerID="fd5890661ee62ac9462d31a77215b6f2ca7ff5c708beca4cb663aaaec083dc74" exitCode=60 Oct 31 00:00:04.909227 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.909194 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" event={"ID":"1e93adf3-6136-459e-89cf-0006f8951204","Type":"ContainerDied","Data":"fd5890661ee62ac9462d31a77215b6f2ca7ff5c708beca4cb663aaaec083dc74"} Oct 31 00:00:04.909379 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.909365 7570 scope.go:122] "RemoveContainer" containerID="fd5890661ee62ac9462d31a77215b6f2ca7ff5c708beca4cb663aaaec083dc74" Oct 31 00:00:04.909624 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:04.909606 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890080-mvk6z_odh-ai-gateway-infra(1e93adf3-6136-459e-89cf-0006f8951204)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" podUID="1e93adf3-6136-459e-89cf-0006f8951204" Oct 31 00:00:04.911028 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.911012 7570 generic.go:356] "Generic (PLEG): container finished" podID="e4a67598-601d-4304-999f-243f467eb6db" containerID="b170d3de492a86cd130c500f3cf7098ccce143f309ee7c6e90f89dac6780bb80" exitCode=60 Oct 31 00:00:04.911102 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.911062 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" event={"ID":"e4a67598-601d-4304-999f-243f467eb6db","Type":"ContainerDied","Data":"b170d3de492a86cd130c500f3cf7098ccce143f309ee7c6e90f89dac6780bb80"} Oct 31 00:00:04.911231 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.911217 7570 scope.go:122] "RemoveContainer" containerID="b170d3de492a86cd130c500f3cf7098ccce143f309ee7c6e90f89dac6780bb80" Oct 31 00:00:04.911439 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:04.911424 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch_odh-ai-gateway-infra(e4a67598-601d-4304-999f-243f467eb6db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" podUID="e4a67598-601d-4304-999f-243f467eb6db" Oct 31 00:00:04.936099 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.936077 7570 scope.go:122] "RemoveContainer" containerID="bb33632cbf22df96b9eb2ea0f996bc655b953dcc86052951db304968e79770ee" Oct 31 00:00:04.951191 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.951094 7570 scope.go:122] "RemoveContainer" containerID="3d96b9481e016a78e6227e2d1aa0a0cf3e98a03d8ed988b9a2fe2ae75affeae2" Oct 31 00:00:04.992024 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:04.992004 7570 scope.go:122] "RemoveContainer" containerID="80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796" Oct 31 00:00:05.006677 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.006652 7570 scope.go:122] "RemoveContainer" containerID="de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb" Oct 31 00:00:05.039506 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.038364 7570 scope.go:122] "RemoveContainer" containerID="80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796" Oct 31 00:00:05.041825 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:05.039813 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796\": container with ID starting with 80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796 not found: ID does not exist" containerID="80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796" Oct 31 00:00:05.041825 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.039846 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796"} err="failed to get container status \"80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796\": rpc error: code = NotFound desc = could not find container \"80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796\": container with ID starting with 80a63286bec2bdd4017488d38f950825bc152e25f8755a8dfeef1dc1b16e1796 not found: ID does not exist" Oct 31 00:00:05.041825 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.039863 7570 scope.go:122] "RemoveContainer" containerID="de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb" Oct 31 00:00:05.041825 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:05.040165 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb\": container with ID starting with de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb not found: ID does not exist" containerID="de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb" Oct 31 00:00:05.041825 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.040197 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb"} err="failed to get container status \"de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb\": rpc error: code = NotFound desc = could not find container \"de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb\": container with ID starting with de3c4ed8bdf39df278f8e3a635767f36fda1161fbb5e77848b4cf53e73e595fb not found: ID does not exist" Oct 31 00:00:05.041825 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.040220 7570 scope.go:122] "RemoveContainer" containerID="f639954ade36bf5010db5dcf04400b4bc7f81de7d235ed2930a1f957a0f2518e" Oct 31 00:00:05.069166 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.064715 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg"] Oct 31 00:00:05.069166 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.065453 7570 scope.go:122] "RemoveContainer" containerID="ad990ea5e755cc9acace5f23ff10441f15e0660023ace9dabd050fe8d9188345" Oct 31 00:00:05.117220 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.117174 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5"] Oct 31 00:00:05.139252 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.139227 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 31 00:00:05.148535 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.148512 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69"] Oct 31 00:00:05.153691 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.153668 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-32732d-openshift-default-5ddfff86b7-2pp69"] Oct 31 00:00:05.195013 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.194987 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-plugins-config-volume\") pod \"201fbcb8-8fed-44e3-b60d-4b89d8c114ca\" (UID: \"201fbcb8-8fed-44e3-b60d-4b89d8c114ca\") " Oct 31 00:00:05.195573 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.195123 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-kube-api-access-65426\" (UniqueName: \"kubernetes.io/projected/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-kube-api-access-65426\") pod \"201fbcb8-8fed-44e3-b60d-4b89d8c114ca\" (UID: \"201fbcb8-8fed-44e3-b60d-4b89d8c114ca\") " Oct 31 00:00:05.195573 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.195414 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-plugins-config-volume" pod "201fbcb8-8fed-44e3-b60d-4b89d8c114ca" (UID: "201fbcb8-8fed-44e3-b60d-4b89d8c114ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:05.199436 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.197052 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:05.199436 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.198063 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-kube-api-access-65426" pod "201fbcb8-8fed-44e3-b60d-4b89d8c114ca" (UID: "201fbcb8-8fed-44e3-b60d-4b89d8c114ca"). InnerVolumeSpecName "kube-api-access-65426". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:05.299149 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.299110 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65426\" (UniqueName: \"kubernetes.io/projected/201fbcb8-8fed-44e3-b60d-4b89d8c114ca-kube-api-access-65426\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:05.745737 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.745706 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs"] Oct 31 00:00:05.746342 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746327 7570 cpu_manager.go:414] "containerMap: removing container" podUID="201fbcb8-8fed-44e3-b60d-4b89d8c114ca" containerName="payload-processing" Oct 31 00:00:05.746412 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746344 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201fbcb8-8fed-44e3-b60d-4b89d8c114ca" containerName="payload-processing" Oct 31 00:00:05.746412 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746370 7570 cpu_manager.go:414] "containerMap: removing container" podUID="42fd7f0b-dfc4-4e90-a321-c4a51115daab" containerName="revoke-keys" Oct 31 00:00:05.746412 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746377 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fd7f0b-dfc4-4e90-a321-c4a51115daab" containerName="revoke-keys" Oct 31 00:00:05.746412 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746387 7570 cpu_manager.go:414] "containerMap: removing container" podUID="de608395-f32c-4f76-ad97-988d2feb7ca0" containerName="image-pruner" Oct 31 00:00:05.746412 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746392 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de608395-f32c-4f76-ad97-988d2feb7ca0" containerName="image-pruner" Oct 31 00:00:05.746412 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746398 7570 cpu_manager.go:414] "containerMap: removing container" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" Oct 31 00:00:05.746412 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746402 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" Oct 31 00:00:05.746412 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746409 7570 cpu_manager.go:414] "containerMap: removing container" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" Oct 31 00:00:05.746412 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746414 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" Oct 31 00:00:05.746687 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746532 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fd7f0b-dfc4-4e90-a321-c4a51115daab" containerName="revoke-keys" Oct 31 00:00:05.746687 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746544 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="de608395-f32c-4f76-ad97-988d2feb7ca0" containerName="image-pruner" Oct 31 00:00:05.746687 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746552 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" Oct 31 00:00:05.746687 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746559 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" containerName="istio-proxy" Oct 31 00:00:05.746687 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.746572 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="201fbcb8-8fed-44e3-b60d-4b89d8c114ca" containerName="payload-processing" Oct 31 00:00:05.750184 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.750166 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:05.754812 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.754742 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:00:05.760883 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.760861 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs"] Oct 31 00:00:05.808839 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.808816 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs\" (UID: \"26a60236-f8f2-4188-b6a4-c74ceeb09143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:05.809007 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.808991 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs\" (UID: \"26a60236-f8f2-4188-b6a4-c74ceeb09143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:05.809077 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.809067 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bkt7\" (UniqueName: \"kubernetes.io/projected/26a60236-f8f2-4188-b6a4-c74ceeb09143-kube-api-access-5bkt7\") pod \"maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs\" (UID: \"26a60236-f8f2-4188-b6a4-c74ceeb09143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:05.910630 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.910605 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bkt7\" (UniqueName: \"kubernetes.io/projected/26a60236-f8f2-4188-b6a4-c74ceeb09143-kube-api-access-5bkt7\") pod \"maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs\" (UID: \"26a60236-f8f2-4188-b6a4-c74ceeb09143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:05.910731 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.910664 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs\" (UID: \"26a60236-f8f2-4188-b6a4-c74ceeb09143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:05.910783 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.910773 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs\" (UID: \"26a60236-f8f2-4188-b6a4-c74ceeb09143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:05.911375 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.911358 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs\" (UID: \"26a60236-f8f2-4188-b6a4-c74ceeb09143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:05.913191 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.913153 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs\" (UID: \"26a60236-f8f2-4188-b6a4-c74ceeb09143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:05.921780 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.921764 7570 scope.go:122] "RemoveContainer" containerID="fd5890661ee62ac9462d31a77215b6f2ca7ff5c708beca4cb663aaaec083dc74" Oct 31 00:00:05.922052 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:05.922032 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890080-mvk6z_odh-ai-gateway-infra(1e93adf3-6136-459e-89cf-0006f8951204)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" podUID="1e93adf3-6136-459e-89cf-0006f8951204" Oct 31 00:00:05.923421 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.923400 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bkt7\" (UniqueName: \"kubernetes.io/projected/26a60236-f8f2-4188-b6a4-c74ceeb09143-kube-api-access-5bkt7\") pod \"maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs\" (UID: \"26a60236-f8f2-4188-b6a4-c74ceeb09143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:05.923822 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.923807 7570 scope.go:122] "RemoveContainer" containerID="b170d3de492a86cd130c500f3cf7098ccce143f309ee7c6e90f89dac6780bb80" Oct 31 00:00:05.924058 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:05.924038 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch_odh-ai-gateway-infra(e4a67598-601d-4304-999f-243f467eb6db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" podUID="e4a67598-601d-4304-999f-243f467eb6db" Oct 31 00:00:05.925006 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.924982 7570 generic.go:356] "Generic (PLEG): container finished" podID="201fbcb8-8fed-44e3-b60d-4b89d8c114ca" containerID="a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f" exitCode=0 Oct 31 00:00:05.925090 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.925034 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" Oct 31 00:00:05.925090 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.925057 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" event={"ID":"201fbcb8-8fed-44e3-b60d-4b89d8c114ca","Type":"ContainerDied","Data":"a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f"} Oct 31 00:00:05.925217 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.925104 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq" event={"ID":"201fbcb8-8fed-44e3-b60d-4b89d8c114ca","Type":"ContainerDied","Data":"5c82ff100bdb042a1b38dd79422ee4ac0e1b8885202939592b84bb7240f60de5"} Oct 31 00:00:05.925217 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.925126 7570 scope.go:122] "RemoveContainer" containerID="a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f" Oct 31 00:00:05.928060 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.928041 7570 scope.go:122] "RemoveContainer" containerID="24dd4fe118cdc74eafaec2e9b78ea0e99bd4ab9ac38bf1ae4988b35178106d5a" Oct 31 00:00:05.928333 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:05.928293 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh_odh-ai-gateway-infra(6b37651e-9dd9-4294-afa7-0bd3f427848c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" Oct 31 00:00:05.929969 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.929954 7570 scope.go:122] "RemoveContainer" containerID="fda1213b39a77702c54d081a4092b797b99e6abbceec0c1cb3dbf96595428705" Oct 31 00:00:05.930361 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:05.930166 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp_odh-ai-gateway-infra(c42b94b3-7dc2-465f-9e1a-f128b015e5b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" podUID="c42b94b3-7dc2-465f-9e1a-f128b015e5b8" Oct 31 00:00:05.931688 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.931672 7570 scope.go:122] "RemoveContainer" containerID="921986a36f6412b4724e6ee00833ab74fbf63002f2ac3a8b3774b330fafb51f9" Oct 31 00:00:05.931880 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:05.931864 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq_odh-ai-gateway-infra(02ffff16-0e62-47a6-9c29-4ad63ebbce79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" Oct 31 00:00:05.951447 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.951431 7570 scope.go:122] "RemoveContainer" containerID="a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f" Oct 31 00:00:05.951811 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:05.951790 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f\": container with ID starting with a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f not found: ID does not exist" containerID="a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f" Oct 31 00:00:05.951888 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:05.951821 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f"} err="failed to get container status \"a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f\": rpc error: code = NotFound desc = could not find container \"a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f\": container with ID starting with a42288bc3280dbdd68a59ef05c61964432def4f6b9be408beda78298d7f20a2f not found: ID does not exist" Oct 31 00:00:06.069057 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:06.068978 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:06.076351 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:06.076326 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq"] Oct 31 00:00:06.079932 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:06.079912 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-200408-d8dc78dd5-xwcpq"] Oct 31 00:00:06.378381 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:00:06.378348 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a60236_f8f2_4188_b6a4_c74ceeb09143.slice/crio-e975104ce737fe9ff88387837639f5373623473bb3694f75811722a09f4e52f9 WatchSource:0}: Error finding container e975104ce737fe9ff88387837639f5373623473bb3694f75811722a09f4e52f9: Status 404 returned error can't find the container with id e975104ce737fe9ff88387837639f5373623473bb3694f75811722a09f4e52f9 Oct 31 00:00:06.379576 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:06.379556 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs"] Oct 31 00:00:06.605484 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:06.605430 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201fbcb8-8fed-44e3-b60d-4b89d8c114ca" path="/var/lib/kubelet/pods/201fbcb8-8fed-44e3-b60d-4b89d8c114ca/volumes" Oct 31 00:00:06.606370 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:06.606346 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a7842f-6043-4b93-ab32-d25e2ba8c5bb" path="/var/lib/kubelet/pods/84a7842f-6043-4b93-ab32-d25e2ba8c5bb/volumes" Oct 31 00:00:06.944168 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:06.944129 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" event={"ID":"26a60236-f8f2-4188-b6a4-c74ceeb09143","Type":"ContainerStarted","Data":"4f9993d222c0b9881fb05b9ec50d101dd246b4fbe13fc95f97d48f2bea530a7f"} Oct 31 00:00:06.944322 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:06.944212 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" event={"ID":"26a60236-f8f2-4188-b6a4-c74ceeb09143","Type":"ContainerStarted","Data":"e975104ce737fe9ff88387837639f5373623473bb3694f75811722a09f4e52f9"} Oct 31 00:00:06.974695 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:06.974649 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" podStartSLOduration=1.974635315 podStartE2EDuration="1.974635315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:06.97076543 +0000 UTC m=+2418.489766520" watchObservedRunningTime="2026-10-31 00:00:06.974635315 +0000 UTC m=+2418.493636400" Oct 31 00:00:07.956541 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:07.953219 7570 generic.go:356] "Generic (PLEG): container finished" podID="26a60236-f8f2-4188-b6a4-c74ceeb09143" containerID="4f9993d222c0b9881fb05b9ec50d101dd246b4fbe13fc95f97d48f2bea530a7f" exitCode=0 Oct 31 00:00:07.956541 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:07.953261 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" event={"ID":"26a60236-f8f2-4188-b6a4-c74ceeb09143","Type":"ContainerDied","Data":"4f9993d222c0b9881fb05b9ec50d101dd246b4fbe13fc95f97d48f2bea530a7f"} Oct 31 00:00:09.117678 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.117654 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:09.155857 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.155832 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-openshift-ca\") pod \"26a60236-f8f2-4188-b6a4-c74ceeb09143\" (UID: \"26a60236-f8f2-4188-b6a4-c74ceeb09143\") " Oct 31 00:00:09.155993 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.155963 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a60236-f8f2-4188-b6a4-c74ceeb09143-kube-api-access-5bkt7\" (UniqueName: \"kubernetes.io/projected/26a60236-f8f2-4188-b6a4-c74ceeb09143-kube-api-access-5bkt7\") pod \"26a60236-f8f2-4188-b6a4-c74ceeb09143\" (UID: \"26a60236-f8f2-4188-b6a4-c74ceeb09143\") " Oct 31 00:00:09.156197 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.156178 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-xks-ca\") pod \"26a60236-f8f2-4188-b6a4-c74ceeb09143\" (UID: \"26a60236-f8f2-4188-b6a4-c74ceeb09143\") " Oct 31 00:00:09.156238 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.156208 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-openshift-ca" pod "26a60236-f8f2-4188-b6a4-c74ceeb09143" (UID: "26a60236-f8f2-4188-b6a4-c74ceeb09143"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:09.156930 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.156903 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:09.158691 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.158658 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a60236-f8f2-4188-b6a4-c74ceeb09143-kube-api-access-5bkt7" pod "26a60236-f8f2-4188-b6a4-c74ceeb09143" (UID: "26a60236-f8f2-4188-b6a4-c74ceeb09143"). InnerVolumeSpecName "kube-api-access-5bkt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:09.158798 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.158723 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-xks-ca" pod "26a60236-f8f2-4188-b6a4-c74ceeb09143" (UID: "26a60236-f8f2-4188-b6a4-c74ceeb09143"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:09.252543 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.252446 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq"] Oct 31 00:00:09.253137 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.253119 7570 cpu_manager.go:414] "containerMap: removing container" podUID="26a60236-f8f2-4188-b6a4-c74ceeb09143" containerName="revoke-keys" Oct 31 00:00:09.253137 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.253136 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a60236-f8f2-4188-b6a4-c74ceeb09143" containerName="revoke-keys" Oct 31 00:00:09.253253 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.253236 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a60236-f8f2-4188-b6a4-c74ceeb09143" containerName="revoke-keys" Oct 31 00:00:09.257043 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.257021 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:09.257943 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.257922 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a60236-f8f2-4188-b6a4-c74ceeb09143-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:09.258006 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.257946 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bkt7\" (UniqueName: \"kubernetes.io/projected/26a60236-f8f2-4188-b6a4-c74ceeb09143-kube-api-access-5bkt7\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:09.275187 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.275161 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq"] Oct 31 00:00:09.359540 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.359490 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d85a55-vs5vq\" (UID: \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:09.359863 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.359841 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d85a55-vs5vq\" (UID: \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:09.359918 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.359882 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qgrp\" (UniqueName: \"kubernetes.io/projected/2ad2904e-63ba-45d4-b7dd-60f0edd67962-kube-api-access-6qgrp\") pod \"maas-api-revoke-sub-79d85a55-vs5vq\" (UID: \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:09.461474 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.461434 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d85a55-vs5vq\" (UID: \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:09.461585 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.461487 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qgrp\" (UniqueName: \"kubernetes.io/projected/2ad2904e-63ba-45d4-b7dd-60f0edd67962-kube-api-access-6qgrp\") pod \"maas-api-revoke-sub-79d85a55-vs5vq\" (UID: \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:09.461649 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.461632 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d85a55-vs5vq\" (UID: \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:09.462645 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.462618 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d85a55-vs5vq\" (UID: \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:09.464167 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.464143 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d85a55-vs5vq\" (UID: \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:09.478504 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.478486 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qgrp\" (UniqueName: \"kubernetes.io/projected/2ad2904e-63ba-45d4-b7dd-60f0edd67962-kube-api-access-6qgrp\") pod \"maas-api-revoke-sub-79d85a55-vs5vq\" (UID: \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:09.572929 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.572895 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:09.740971 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.740609 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.772131 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67262faa-4e17-4e80-9d44-917ef37e70db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67262faa-4e17-4e80-9d44-917ef37e70db-istiod-ca-cert\") pod \"67262faa-4e17-4e80-9d44-917ef37e70db\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.772173 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-istio-token\" (UniqueName: \"kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-istio-token\") pod \"67262faa-4e17-4e80-9d44-917ef37e70db\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.772225 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-data\") pod \"67262faa-4e17-4e80-9d44-917ef37e70db\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.772412 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-envoy\") pod \"67262faa-4e17-4e80-9d44-917ef37e70db\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.772476 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-socket\") pod \"67262faa-4e17-4e80-9d44-917ef37e70db\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.772507 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67262faa-4e17-4e80-9d44-917ef37e70db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67262faa-4e17-4e80-9d44-917ef37e70db-istio-podinfo\") pod \"67262faa-4e17-4e80-9d44-917ef37e70db\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.772639 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-certs\") pod \"67262faa-4e17-4e80-9d44-917ef37e70db\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.772828 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-kube-api-access-b2r47\" (UniqueName: \"kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-kube-api-access-b2r47\") pod \"67262faa-4e17-4e80-9d44-917ef37e70db\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.772886 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-credential-socket\") pod \"67262faa-4e17-4e80-9d44-917ef37e70db\" (UID: \"67262faa-4e17-4e80-9d44-917ef37e70db\") " Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.777024 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-certs" pod "67262faa-4e17-4e80-9d44-917ef37e70db" (UID: "67262faa-4e17-4e80-9d44-917ef37e70db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.777107 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-data" pod "67262faa-4e17-4e80-9d44-917ef37e70db" (UID: "67262faa-4e17-4e80-9d44-917ef37e70db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.777564 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-credential-socket" pod "67262faa-4e17-4e80-9d44-917ef37e70db" (UID: "67262faa-4e17-4e80-9d44-917ef37e70db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.779553 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67262faa-4e17-4e80-9d44-917ef37e70db-istiod-ca-cert" pod "67262faa-4e17-4e80-9d44-917ef37e70db" (UID: "67262faa-4e17-4e80-9d44-917ef37e70db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:09.781566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.779601 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-socket" pod "67262faa-4e17-4e80-9d44-917ef37e70db" (UID: "67262faa-4e17-4e80-9d44-917ef37e70db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:09.782935 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.782910 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-envoy" pod "67262faa-4e17-4e80-9d44-917ef37e70db" (UID: "67262faa-4e17-4e80-9d44-917ef37e70db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:09.784086 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.784057 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-kube-api-access-b2r47" pod "67262faa-4e17-4e80-9d44-917ef37e70db" (UID: "67262faa-4e17-4e80-9d44-917ef37e70db"). InnerVolumeSpecName "kube-api-access-b2r47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:09.785206 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.785178 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67262faa-4e17-4e80-9d44-917ef37e70db-istio-podinfo" pod "67262faa-4e17-4e80-9d44-917ef37e70db" (UID: "67262faa-4e17-4e80-9d44-917ef37e70db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:00:09.786298 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.786274 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-istio-token" pod "67262faa-4e17-4e80-9d44-917ef37e70db" (UID: "67262faa-4e17-4e80-9d44-917ef37e70db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:09.875754 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.875710 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:09.875754 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.875748 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:09.875754 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.875759 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67262faa-4e17-4e80-9d44-917ef37e70db-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:09.875754 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.875768 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:09.875980 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.875777 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2r47\" (UniqueName: \"kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-kube-api-access-b2r47\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:09.875980 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.875787 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:09.875980 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.875797 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67262faa-4e17-4e80-9d44-917ef37e70db-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:09.875980 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.875805 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67262faa-4e17-4e80-9d44-917ef37e70db-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:09.875980 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.875813 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67262faa-4e17-4e80-9d44-917ef37e70db-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:09.938264 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.938236 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq"] Oct 31 00:00:09.939553 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:00:09.939523 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad2904e_63ba_45d4_b7dd_60f0edd67962.slice/crio-df5ca0473d0922c42b2a75e009c4f3b7ec56eaf19a943130c486a31f3e95ef5e WatchSource:0}: Error finding container df5ca0473d0922c42b2a75e009c4f3b7ec56eaf19a943130c486a31f3e95ef5e: Status 404 returned error can't find the container with id df5ca0473d0922c42b2a75e009c4f3b7ec56eaf19a943130c486a31f3e95ef5e Oct 31 00:00:09.996448 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.996408 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" Oct 31 00:00:09.996448 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.996402 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs" event={"ID":"26a60236-f8f2-4188-b6a4-c74ceeb09143","Type":"ContainerDied","Data":"e975104ce737fe9ff88387837639f5373623473bb3694f75811722a09f4e52f9"} Oct 31 00:00:09.996686 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.996561 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e975104ce737fe9ff88387837639f5373623473bb3694f75811722a09f4e52f9" Oct 31 00:00:09.998182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.998152 7570 generic.go:356] "Generic (PLEG): container finished" podID="67262faa-4e17-4e80-9d44-917ef37e70db" containerID="5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357" exitCode=0 Oct 31 00:00:09.998296 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.998180 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" event={"ID":"67262faa-4e17-4e80-9d44-917ef37e70db","Type":"ContainerDied","Data":"5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357"} Oct 31 00:00:09.998296 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.998215 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" Oct 31 00:00:09.998296 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.998236 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq" event={"ID":"67262faa-4e17-4e80-9d44-917ef37e70db","Type":"ContainerDied","Data":"bb08f631d7bd8e1e107d53fa8ae79e6a1fa994d2dcea9671bb9d0804102e06bf"} Oct 31 00:00:09.998296 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:09.998265 7570 scope.go:122] "RemoveContainer" containerID="5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357" Oct 31 00:00:10.000432 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.000403 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" event={"ID":"2ad2904e-63ba-45d4-b7dd-60f0edd67962","Type":"ContainerStarted","Data":"df5ca0473d0922c42b2a75e009c4f3b7ec56eaf19a943130c486a31f3e95ef5e"} Oct 31 00:00:10.013629 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.012535 7570 scope.go:122] "RemoveContainer" containerID="5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357" Oct 31 00:00:10.013629 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:10.012874 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357\": container with ID starting with 5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357 not found: ID does not exist" containerID="5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357" Oct 31 00:00:10.013629 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.012905 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357"} err="failed to get container status \"5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357\": rpc error: code = NotFound desc = could not find container \"5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357\": container with ID starting with 5f6074968b0a79f6a15d3bdfc724fc4dfe6052398653982128c0fa624fad0357 not found: ID does not exist" Oct 31 00:00:10.044687 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.043111 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq"] Oct 31 00:00:10.090486 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.090289 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" podUID="7f016eec-7091-4a3e-847e-7a382085ffef" containerName="payload-processing" containerID="cri-o://65b340054ff204ea3e9089ea510d55bf4e3712e10df9dd29f7782872c7f0b071" gracePeriod=25 Oct 31 00:00:10.135484 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.125530 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" podUID="c070c544-43c5-4796-b385-6c4094a63624" containerName="payload-pre-processing" containerID="cri-o://d2d0452178b2079a293c0673a10ebba578c88a88d0facf27ff846b2140edd577" gracePeriod=25 Oct 31 00:00:10.135484 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.125819 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-687rq"] Oct 31 00:00:10.604079 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.604041 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" path="/var/lib/kubelet/pods/67262faa-4e17-4e80-9d44-917ef37e70db/volumes" Oct 31 00:00:10.739724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.739653 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f"] Oct 31 00:00:10.997759 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.997683 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd"] Oct 31 00:00:10.998553 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.998534 7570 cpu_manager.go:414] "containerMap: removing container" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" Oct 31 00:00:10.998553 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.998554 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" Oct 31 00:00:10.998778 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:10.998764 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="67262faa-4e17-4e80-9d44-917ef37e70db" containerName="istio-proxy" Oct 31 00:00:11.002942 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.002921 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.012081 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.012059 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" event={"ID":"2ad2904e-63ba-45d4-b7dd-60f0edd67962","Type":"ContainerStarted","Data":"55411c1657d957f452c90681a8c1bc2120c505043443a28cc6be687ffe62d5a9"} Oct 31 00:00:11.016719 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.016681 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-200408-openshift-default-dockercfg-rzm4l\"" Oct 31 00:00:11.020397 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.020376 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd"] Oct 31 00:00:11.118335 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.118306 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-token\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.118539 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.118340 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istiod-ca-cert\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.118539 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.118497 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-data\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.118539 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.118530 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-envoy\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.118702 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.118552 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-podinfo\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.118702 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.118693 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-credential-socket\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.118876 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.118847 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bkxr\" (UniqueName: \"kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-kube-api-access-5bkxr\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.118972 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.118959 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-certs\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.119033 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.119014 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-socket\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.220358 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.220326 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-token\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.220358 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.220361 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istiod-ca-cert\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.220885 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.220449 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-data\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.220885 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.220510 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-envoy\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.220885 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.220533 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-podinfo\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.220885 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.220591 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-credential-socket\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.220885 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.220728 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bkxr\" (UniqueName: \"kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-kube-api-access-5bkxr\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.220885 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.220812 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-certs\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.220885 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.220840 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-socket\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.221200 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.220959 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-data\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.221200 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.221030 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-credential-socket\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.221200 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.221111 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-certs\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.221531 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.221201 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-socket\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.221531 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.221372 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istiod-ca-cert\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.222952 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.222930 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-envoy\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.223346 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.223303 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-podinfo\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.233353 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.233331 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bkxr\" (UniqueName: \"kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-kube-api-access-5bkxr\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.234095 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.234072 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-token\") pod \"e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.402161 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.402136 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:11.702057 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.702030 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd"] Oct 31 00:00:11.702646 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:00:11.702623 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b57f5ed_88f7_4eb3_a209_3b0e24dd33f8.slice/crio-acc4e95013981419d8feb265350bbaa1f26b4997ff6516bb952280266c101fc7 WatchSource:0}: Error finding container acc4e95013981419d8feb265350bbaa1f26b4997ff6516bb952280266c101fc7: Status 404 returned error can't find the container with id acc4e95013981419d8feb265350bbaa1f26b4997ff6516bb952280266c101fc7 Oct 31 00:00:11.705351 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.705321 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:00:11.705413 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.705391 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:00:11.705444 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:11.705423 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:00:12.021813 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.021720 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" event={"ID":"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8","Type":"ContainerStarted","Data":"13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21"} Oct 31 00:00:12.021813 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.021782 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" event={"ID":"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8","Type":"ContainerStarted","Data":"acc4e95013981419d8feb265350bbaa1f26b4997ff6516bb952280266c101fc7"} Oct 31 00:00:12.023564 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.023537 7570 generic.go:356] "Generic (PLEG): container finished" podID="2ad2904e-63ba-45d4-b7dd-60f0edd67962" containerID="55411c1657d957f452c90681a8c1bc2120c505043443a28cc6be687ffe62d5a9" exitCode=0 Oct 31 00:00:12.023686 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.023620 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" event={"ID":"2ad2904e-63ba-45d4-b7dd-60f0edd67962","Type":"ContainerDied","Data":"55411c1657d957f452c90681a8c1bc2120c505043443a28cc6be687ffe62d5a9"} Oct 31 00:00:12.050200 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.050149 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" podStartSLOduration=2.050135925 podStartE2EDuration="2.050135925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:12.047761784 +0000 UTC m=+2423.566762874" watchObservedRunningTime="2026-10-31 00:00:12.050135925 +0000 UTC m=+2423.569137013" Oct 31 00:00:12.178631 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.178608 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:12.334887 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.334808 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad2904e-63ba-45d4-b7dd-60f0edd67962-kube-api-access-6qgrp\" (UniqueName: \"kubernetes.io/projected/2ad2904e-63ba-45d4-b7dd-60f0edd67962-kube-api-access-6qgrp\") pod \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\" (UID: \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\") " Oct 31 00:00:12.335383 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.335028 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-openshift-ca\") pod \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\" (UID: \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\") " Oct 31 00:00:12.335383 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.335084 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-xks-ca\") pod \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\" (UID: \"2ad2904e-63ba-45d4-b7dd-60f0edd67962\") " Oct 31 00:00:12.336446 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.336399 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-openshift-ca" pod "2ad2904e-63ba-45d4-b7dd-60f0edd67962" (UID: "2ad2904e-63ba-45d4-b7dd-60f0edd67962"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:12.337601 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.337573 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad2904e-63ba-45d4-b7dd-60f0edd67962-kube-api-access-6qgrp" pod "2ad2904e-63ba-45d4-b7dd-60f0edd67962" (UID: "2ad2904e-63ba-45d4-b7dd-60f0edd67962"). InnerVolumeSpecName "kube-api-access-6qgrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:12.338409 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.338382 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-xks-ca" pod "2ad2904e-63ba-45d4-b7dd-60f0edd67962" (UID: "2ad2904e-63ba-45d4-b7dd-60f0edd67962"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:12.402404 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.402379 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:12.404151 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.404129 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 31 00:00:12.404235 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.404171 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" podUID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 31 00:00:12.436623 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.436603 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qgrp\" (UniqueName: \"kubernetes.io/projected/2ad2904e-63ba-45d4-b7dd-60f0edd67962-kube-api-access-6qgrp\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:12.436623 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.436623 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:12.436750 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:12.436635 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad2904e-63ba-45d4-b7dd-60f0edd67962-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:13.034042 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:13.033961 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" Oct 31 00:00:13.034042 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:13.034004 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq" event={"ID":"2ad2904e-63ba-45d4-b7dd-60f0edd67962","Type":"ContainerDied","Data":"df5ca0473d0922c42b2a75e009c4f3b7ec56eaf19a943130c486a31f3e95ef5e"} Oct 31 00:00:13.034233 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:13.034051 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df5ca0473d0922c42b2a75e009c4f3b7ec56eaf19a943130c486a31f3e95ef5e" Oct 31 00:00:13.135614 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:13.135021 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" podUID="7f016eec-7091-4a3e-847e-7a382085ffef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:00:13.176734 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:13.176700 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" podUID="c070c544-43c5-4796-b385-6c4094a63624" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:00:13.424898 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:13.424855 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 31 00:00:13.425399 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:13.424913 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" podUID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 31 00:00:14.034283 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.034242 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" containerID="cri-o://c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146" gracePeriod=45 Oct 31 00:00:14.038530 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.038494 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" containerID="cri-o://92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8" gracePeriod=45 Oct 31 00:00:14.039998 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.039970 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": read tcp 10.134.0.2:55832->10.134.0.70:8000: read: connection reset by peer" Oct 31 00:00:14.044547 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.044522 7570 generic.go:356] "Generic (PLEG): container finished" podID="ed360b42-1a02-4474-9bf1-2117d1b438f1" containerID="d6c071d29384152b592df801487206c5ca8e68ec1e11a4f296680f8452dffa8d" exitCode=0 Oct 31 00:00:14.044623 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.044604 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" event={"ID":"ed360b42-1a02-4474-9bf1-2117d1b438f1","Type":"ContainerDied","Data":"d6c071d29384152b592df801487206c5ca8e68ec1e11a4f296680f8452dffa8d"} Oct 31 00:00:14.044665 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.044657 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" event={"ID":"ed360b42-1a02-4474-9bf1-2117d1b438f1","Type":"ContainerStarted","Data":"671a9cc22af912bda6dbadbc6d91ef8a54415c0c3e280620cf61901099729eb8"} Oct 31 00:00:14.044735 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.044704 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 31 00:00:14.052898 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.052877 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" containerID="cri-o://72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98" gracePeriod=45 Oct 31 00:00:14.166157 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.166118 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused" Oct 31 00:00:14.317988 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.317942 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": dial tcp 10.134.0.81:8000: connect: connection refused" Oct 31 00:00:14.424667 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.409834 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 31 00:00:14.424667 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.409896 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" podUID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 31 00:00:14.571711 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.571625 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 31 00:00:14.636623 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.634652 7570 scope.go:122] "RemoveContainer" containerID="fd5890661ee62ac9462d31a77215b6f2ca7ff5c708beca4cb663aaaec083dc74" Oct 31 00:00:14.654556 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.654027 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 31 00:00:14.657595 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.657574 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 31 00:00:14.668414 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.668265 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f1bda07-310d-43a8-a926-9004487b0034-kube-api-access-6wgvm\" (UniqueName: \"kubernetes.io/projected/4f1bda07-310d-43a8-a926-9004487b0034-kube-api-access-6wgvm\") pod \"4f1bda07-310d-43a8-a926-9004487b0034\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " Oct 31 00:00:14.668414 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.668391 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-dshm\") pod \"4f1bda07-310d-43a8-a926-9004487b0034\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " Oct 31 00:00:14.668652 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.668501 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f1bda07-310d-43a8-a926-9004487b0034-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1bda07-310d-43a8-a926-9004487b0034-tls-certs\") pod \"4f1bda07-310d-43a8-a926-9004487b0034\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " Oct 31 00:00:14.668652 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.668627 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-home\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-home\") pod \"4f1bda07-310d-43a8-a926-9004487b0034\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " Oct 31 00:00:14.669660 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.669540 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-tmp-dir\") pod \"4f1bda07-310d-43a8-a926-9004487b0034\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " Oct 31 00:00:14.669660 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.669597 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-model-cache\") pod \"4f1bda07-310d-43a8-a926-9004487b0034\" (UID: \"4f1bda07-310d-43a8-a926-9004487b0034\") " Oct 31 00:00:14.670892 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.670861 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-home" pod "4f1bda07-310d-43a8-a926-9004487b0034" (UID: "4f1bda07-310d-43a8-a926-9004487b0034"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:14.671298 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.671084 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-tmp-dir" pod "4f1bda07-310d-43a8-a926-9004487b0034" (UID: "4f1bda07-310d-43a8-a926-9004487b0034"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:14.672252 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.672192 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-model-cache" pod "4f1bda07-310d-43a8-a926-9004487b0034" (UID: "4f1bda07-310d-43a8-a926-9004487b0034"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:14.674732 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.674693 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-dshm" pod "4f1bda07-310d-43a8-a926-9004487b0034" (UID: "4f1bda07-310d-43a8-a926-9004487b0034"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:14.676161 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.676139 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1bda07-310d-43a8-a926-9004487b0034-kube-api-access-6wgvm" pod "4f1bda07-310d-43a8-a926-9004487b0034" (UID: "4f1bda07-310d-43a8-a926-9004487b0034"). InnerVolumeSpecName "kube-api-access-6wgvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:14.678365 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.678346 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f1bda07-310d-43a8-a926-9004487b0034-tls-certs" pod "4f1bda07-310d-43a8-a926-9004487b0034" (UID: "4f1bda07-310d-43a8-a926-9004487b0034"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:14.743614 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.743511 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq"] Oct 31 00:00:14.750309 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.750279 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-200408-2wjrq"] Oct 31 00:00:14.771630 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.771602 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-home\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-home\") pod \"d073cc7c-6856-4131-8bf7-250400f05d7f\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " Oct 31 00:00:14.771781 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.771643 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d073cc7c-6856-4131-8bf7-250400f05d7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/d073cc7c-6856-4131-8bf7-250400f05d7f-tls-certs\") pod \"d073cc7c-6856-4131-8bf7-250400f05d7f\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " Oct 31 00:00:14.771781 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.771698 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-home\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-home\") pod \"80dc7b17-04ea-4d03-a843-24891d3cda07\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " Oct 31 00:00:14.771781 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.771729 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-tmp-dir\") pod \"d073cc7c-6856-4131-8bf7-250400f05d7f\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " Oct 31 00:00:14.771781 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.771767 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80dc7b17-04ea-4d03-a843-24891d3cda07-tls-certs\" (UniqueName: \"kubernetes.io/secret/80dc7b17-04ea-4d03-a843-24891d3cda07-tls-certs\") pod \"80dc7b17-04ea-4d03-a843-24891d3cda07\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " Oct 31 00:00:14.771991 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.771890 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-model-cache\") pod \"d073cc7c-6856-4131-8bf7-250400f05d7f\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " Oct 31 00:00:14.771991 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.771920 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-home" pod "d073cc7c-6856-4131-8bf7-250400f05d7f" (UID: "d073cc7c-6856-4131-8bf7-250400f05d7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:14.772089 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.771991 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d073cc7c-6856-4131-8bf7-250400f05d7f-kube-api-access-tpngw\" (UniqueName: \"kubernetes.io/projected/d073cc7c-6856-4131-8bf7-250400f05d7f-kube-api-access-tpngw\") pod \"d073cc7c-6856-4131-8bf7-250400f05d7f\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " Oct 31 00:00:14.772089 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.772078 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-dshm\") pod \"80dc7b17-04ea-4d03-a843-24891d3cda07\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " Oct 31 00:00:14.772186 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.772115 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dc7b17-04ea-4d03-a843-24891d3cda07-kube-api-access-shxk5\" (UniqueName: \"kubernetes.io/projected/80dc7b17-04ea-4d03-a843-24891d3cda07-kube-api-access-shxk5\") pod \"80dc7b17-04ea-4d03-a843-24891d3cda07\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " Oct 31 00:00:14.772186 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.772118 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-tmp-dir" pod "d073cc7c-6856-4131-8bf7-250400f05d7f" (UID: "d073cc7c-6856-4131-8bf7-250400f05d7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:14.772406 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.772377 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-model-cache" pod "d073cc7c-6856-4131-8bf7-250400f05d7f" (UID: "d073cc7c-6856-4131-8bf7-250400f05d7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:14.772825 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.772764 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-home" pod "80dc7b17-04ea-4d03-a843-24891d3cda07" (UID: "80dc7b17-04ea-4d03-a843-24891d3cda07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:14.773420 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.773401 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-model-cache\") pod \"80dc7b17-04ea-4d03-a843-24891d3cda07\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " Oct 31 00:00:14.773993 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.773876 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-model-cache" pod "80dc7b17-04ea-4d03-a843-24891d3cda07" (UID: "80dc7b17-04ea-4d03-a843-24891d3cda07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:14.774150 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.774131 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-dshm\") pod \"d073cc7c-6856-4131-8bf7-250400f05d7f\" (UID: \"d073cc7c-6856-4131-8bf7-250400f05d7f\") " Oct 31 00:00:14.774630 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.774613 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-tmp-dir\") pod \"80dc7b17-04ea-4d03-a843-24891d3cda07\" (UID: \"80dc7b17-04ea-4d03-a843-24891d3cda07\") " Oct 31 00:00:14.775066 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.774951 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-tmp-dir" pod "80dc7b17-04ea-4d03-a843-24891d3cda07" (UID: "80dc7b17-04ea-4d03-a843-24891d3cda07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:14.775979 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.775954 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wgvm\" (UniqueName: \"kubernetes.io/projected/4f1bda07-310d-43a8-a926-9004487b0034-kube-api-access-6wgvm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.776119 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.776094 7570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-model-cache\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.776198 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.776121 7570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-dshm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.776198 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.776136 7570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-tmp-dir\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.776198 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.776153 7570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1bda07-310d-43a8-a926-9004487b0034-tls-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.776198 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.776166 7570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-home\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.776198 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.776178 7570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-home\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.776198 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.776190 7570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-tmp-dir\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.776385 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.776203 7570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-home\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.776385 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.776216 7570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-tmp-dir\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.776385 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.776228 7570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1bda07-310d-43a8-a926-9004487b0034-model-cache\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.776385 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.776243 7570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-model-cache\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.812626 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.812594 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d073cc7c-6856-4131-8bf7-250400f05d7f-tls-certs" pod "d073cc7c-6856-4131-8bf7-250400f05d7f" (UID: "d073cc7c-6856-4131-8bf7-250400f05d7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:14.812626 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.812606 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80dc7b17-04ea-4d03-a843-24891d3cda07-tls-certs" pod "80dc7b17-04ea-4d03-a843-24891d3cda07" (UID: "80dc7b17-04ea-4d03-a843-24891d3cda07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:14.813603 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.813578 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-dshm" pod "80dc7b17-04ea-4d03-a843-24891d3cda07" (UID: "80dc7b17-04ea-4d03-a843-24891d3cda07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:14.813716 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.813702 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d073cc7c-6856-4131-8bf7-250400f05d7f-kube-api-access-tpngw" pod "d073cc7c-6856-4131-8bf7-250400f05d7f" (UID: "d073cc7c-6856-4131-8bf7-250400f05d7f"). InnerVolumeSpecName "kube-api-access-tpngw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:14.814147 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.814115 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dc7b17-04ea-4d03-a843-24891d3cda07-kube-api-access-shxk5" pod "80dc7b17-04ea-4d03-a843-24891d3cda07" (UID: "80dc7b17-04ea-4d03-a843-24891d3cda07"). InnerVolumeSpecName "kube-api-access-shxk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:14.814490 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.814439 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-dshm" pod "d073cc7c-6856-4131-8bf7-250400f05d7f" (UID: "d073cc7c-6856-4131-8bf7-250400f05d7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:14.877315 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.877217 7570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d073cc7c-6856-4131-8bf7-250400f05d7f-dshm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.877315 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.877244 7570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d073cc7c-6856-4131-8bf7-250400f05d7f-tls-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.877315 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.877253 7570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80dc7b17-04ea-4d03-a843-24891d3cda07-tls-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.877315 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.877276 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpngw\" (UniqueName: \"kubernetes.io/projected/d073cc7c-6856-4131-8bf7-250400f05d7f-kube-api-access-tpngw\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.877315 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.877285 7570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80dc7b17-04ea-4d03-a843-24891d3cda07-dshm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:14.877315 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:14.877294 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shxk5\" (UniqueName: \"kubernetes.io/projected/80dc7b17-04ea-4d03-a843-24891d3cda07-kube-api-access-shxk5\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:15.054677 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.054641 7570 generic.go:356] "Generic (PLEG): container finished" podID="4f1bda07-310d-43a8-a926-9004487b0034" containerID="92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8" exitCode=0 Oct 31 00:00:15.055710 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.054853 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" Oct 31 00:00:15.055710 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.054861 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" event={"ID":"4f1bda07-310d-43a8-a926-9004487b0034","Type":"ContainerDied","Data":"92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8"} Oct 31 00:00:15.055710 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.054911 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd" event={"ID":"4f1bda07-310d-43a8-a926-9004487b0034","Type":"ContainerDied","Data":"9156f3c108220aa31db305d5ed5205f257011d4f57f8818644c3dc33791a5458"} Oct 31 00:00:15.055710 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.054935 7570 scope.go:122] "RemoveContainer" containerID="92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8" Oct 31 00:00:15.065497 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.059506 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" Oct 31 00:00:15.065497 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.059517 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" event={"ID":"80dc7b17-04ea-4d03-a843-24891d3cda07","Type":"ContainerDied","Data":"c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146"} Oct 31 00:00:15.065497 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.059420 7570 generic.go:356] "Generic (PLEG): container finished" podID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerID="c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146" exitCode=0 Oct 31 00:00:15.065497 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.060212 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp" event={"ID":"80dc7b17-04ea-4d03-a843-24891d3cda07","Type":"ContainerDied","Data":"91f7af2fe72921d591025cf070d4c1bb77167c54f55300bbb3d8496c87984bfb"} Oct 31 00:00:15.068479 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.067412 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" event={"ID":"1e93adf3-6136-459e-89cf-0006f8951204","Type":"ContainerStarted","Data":"9c61ec523df5324bb5e3925237cd3bfb411eb96e3d2eb11c1fa6732256826d6e"} Oct 31 00:00:15.070029 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.070006 7570 generic.go:356] "Generic (PLEG): container finished" podID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerID="72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98" exitCode=0 Oct 31 00:00:15.070118 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.070062 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" Oct 31 00:00:15.070168 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.070064 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" event={"ID":"d073cc7c-6856-4131-8bf7-250400f05d7f","Type":"ContainerDied","Data":"72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98"} Oct 31 00:00:15.070168 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.070159 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn" event={"ID":"d073cc7c-6856-4131-8bf7-250400f05d7f","Type":"ContainerDied","Data":"b2e2fbb04e1dcc3a710526081b32e9910f1590b40e6e6cce6ecbdc6684ed9241"} Oct 31 00:00:15.085276 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.085262 7570 scope.go:122] "RemoveContainer" containerID="92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8" Oct 31 00:00:15.106494 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:15.102203 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8\": container with ID starting with 92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8 not found: ID does not exist" containerID="92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8" Oct 31 00:00:15.106494 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.102244 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8"} err="failed to get container status \"92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8\": rpc error: code = NotFound desc = could not find container \"92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8\": container with ID starting with 92ace3f5c4271812bd84099a2b42994dc7338a555fbf56292308df9f0bf580b8 not found: ID does not exist" Oct 31 00:00:15.106494 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.102268 7570 scope.go:122] "RemoveContainer" containerID="c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146" Oct 31 00:00:15.134717 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.134687 7570 scope.go:122] "RemoveContainer" containerID="c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146" Oct 31 00:00:15.135057 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:15.135033 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146\": container with ID starting with c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146 not found: ID does not exist" containerID="c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146" Oct 31 00:00:15.135123 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.135067 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146"} err="failed to get container status \"c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146\": rpc error: code = NotFound desc = could not find container \"c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146\": container with ID starting with c3267f590337490e49218f07d49edfa5b8c4cd8b88d9f8b3260de8b764384146 not found: ID does not exist" Oct 31 00:00:15.135123 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.135094 7570 scope.go:122] "RemoveContainer" containerID="72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98" Oct 31 00:00:15.148070 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.147990 7570 scope.go:122] "RemoveContainer" containerID="72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98" Oct 31 00:00:15.148349 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:15.148329 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98\": container with ID starting with 72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98 not found: ID does not exist" containerID="72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98" Oct 31 00:00:15.148434 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.148359 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98"} err="failed to get container status \"72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98\": rpc error: code = NotFound desc = could not find container \"72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98\": container with ID starting with 72b52c56204ebda4bf58be1accb5659d07d18d2b6a39c22f73f180922709ec98 not found: ID does not exist" Oct 31 00:00:15.152007 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.151976 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd"] Oct 31 00:00:15.156708 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.156687 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/premium-w4-843d8c-kserve-7844d66896-27fxd"] Oct 31 00:00:15.171421 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.171398 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn"] Oct 31 00:00:15.179830 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.179798 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/distinct-2-w4-843d8c-kserve-8644666f64-vhwnn"] Oct 31 00:00:15.217569 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.217537 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp"] Oct 31 00:00:15.220806 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.220783 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/distinct-w4-843d8c-kserve-df87d4cff-tz6fp"] Oct 31 00:00:15.410589 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.408810 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 31 00:00:15.410589 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.408869 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" podUID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 31 00:00:15.613435 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.595386 7570 scope.go:122] "RemoveContainer" containerID="24dd4fe118cdc74eafaec2e9b78ea0e99bd4ab9ac38bf1ae4988b35178106d5a" Oct 31 00:00:15.613435 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:15.611028 7570 scope.go:122] "RemoveContainer" containerID="921986a36f6412b4724e6ee00833ab74fbf63002f2ac3a8b3774b330fafb51f9" Oct 31 00:00:16.085281 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:16.085252 7570 generic.go:356] "Generic (PLEG): container finished" podID="1e93adf3-6136-459e-89cf-0006f8951204" containerID="9c61ec523df5324bb5e3925237cd3bfb411eb96e3d2eb11c1fa6732256826d6e" exitCode=60 Oct 31 00:00:16.085415 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:16.085324 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" event={"ID":"1e93adf3-6136-459e-89cf-0006f8951204","Type":"ContainerDied","Data":"9c61ec523df5324bb5e3925237cd3bfb411eb96e3d2eb11c1fa6732256826d6e"} Oct 31 00:00:16.085415 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:16.085389 7570 scope.go:122] "RemoveContainer" containerID="fd5890661ee62ac9462d31a77215b6f2ca7ff5c708beca4cb663aaaec083dc74" Oct 31 00:00:16.124778 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:16.124717 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z"] Oct 31 00:00:16.136896 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:16.136871 7570 scope.go:122] "RemoveContainer" containerID="9c61ec523df5324bb5e3925237cd3bfb411eb96e3d2eb11c1fa6732256826d6e" Oct 31 00:00:16.137395 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:16.137355 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890080-mvk6z_odh-ai-gateway-infra(1e93adf3-6136-459e-89cf-0006f8951204)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" podUID="1e93adf3-6136-459e-89cf-0006f8951204" Oct 31 00:00:16.403848 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:16.403811 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 31 00:00:16.404101 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:16.404074 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" podUID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 31 00:00:16.605496 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:16.603946 7570 scope.go:122] "RemoveContainer" containerID="b170d3de492a86cd130c500f3cf7098ccce143f309ee7c6e90f89dac6780bb80" Oct 31 00:00:16.619178 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:16.619156 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f1bda07-310d-43a8-a926-9004487b0034" path="/var/lib/kubelet/pods/4f1bda07-310d-43a8-a926-9004487b0034/volumes" Oct 31 00:00:16.619703 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:16.619687 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65510995-3f6f-46ad-a509-a253f117b668" path="/var/lib/kubelet/pods/65510995-3f6f-46ad-a509-a253f117b668/volumes" Oct 31 00:00:16.620090 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:16.620078 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" path="/var/lib/kubelet/pods/80dc7b17-04ea-4d03-a843-24891d3cda07/volumes" Oct 31 00:00:16.620508 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:16.620495 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" path="/var/lib/kubelet/pods/d073cc7c-6856-4131-8bf7-250400f05d7f/volumes" Oct 31 00:00:17.131895 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.131817 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" event={"ID":"e4a67598-601d-4304-999f-243f467eb6db","Type":"ContainerStarted","Data":"a70279b6105b8ef22af7b00c7195d1164d05e8e289857c176820bb2e36dfe612"} Oct 31 00:00:17.134773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.134128 7570 generic.go:356] "Generic (PLEG): container finished" podID="6b37651e-9dd9-4294-afa7-0bd3f427848c" containerID="fd74e0af3a0429e9e7f0fd1bc50dbe19a83b154409681f0a114297b2eb0dbdec" exitCode=60 Oct 31 00:00:17.134773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.134227 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" event={"ID":"6b37651e-9dd9-4294-afa7-0bd3f427848c","Type":"ContainerDied","Data":"fd74e0af3a0429e9e7f0fd1bc50dbe19a83b154409681f0a114297b2eb0dbdec"} Oct 31 00:00:17.134773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.134264 7570 scope.go:122] "RemoveContainer" containerID="24dd4fe118cdc74eafaec2e9b78ea0e99bd4ab9ac38bf1ae4988b35178106d5a" Oct 31 00:00:17.134773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.134608 7570 scope.go:122] "RemoveContainer" containerID="fd74e0af3a0429e9e7f0fd1bc50dbe19a83b154409681f0a114297b2eb0dbdec" Oct 31 00:00:17.135366 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:17.135082 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh_odh-ai-gateway-infra(6b37651e-9dd9-4294-afa7-0bd3f427848c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" Oct 31 00:00:17.143501 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.143444 7570 generic.go:356] "Generic (PLEG): container finished" podID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" containerID="95edfc6b7f265f351fece984465b7c6b6f9412299bd9a08a33ab2f0cf93e06eb" exitCode=60 Oct 31 00:00:17.143608 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.143589 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" event={"ID":"02ffff16-0e62-47a6-9c29-4ad63ebbce79","Type":"ContainerDied","Data":"95edfc6b7f265f351fece984465b7c6b6f9412299bd9a08a33ab2f0cf93e06eb"} Oct 31 00:00:17.143950 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.143931 7570 scope.go:122] "RemoveContainer" containerID="95edfc6b7f265f351fece984465b7c6b6f9412299bd9a08a33ab2f0cf93e06eb" Oct 31 00:00:17.144222 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:17.144203 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq_odh-ai-gateway-infra(02ffff16-0e62-47a6-9c29-4ad63ebbce79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" Oct 31 00:00:17.158911 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.158877 7570 scope.go:122] "RemoveContainer" containerID="921986a36f6412b4724e6ee00833ab74fbf63002f2ac3a8b3774b330fafb51f9" Oct 31 00:00:17.352337 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.352308 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" Oct 31 00:00:17.411633 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.405069 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 31 00:00:17.411633 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.405124 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" podUID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 31 00:00:17.450964 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.450885 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e93adf3-6136-459e-89cf-0006f8951204-kube-api-access-49g56\" (UniqueName: \"kubernetes.io/projected/1e93adf3-6136-459e-89cf-0006f8951204-kube-api-access-49g56\") pod \"1e93adf3-6136-459e-89cf-0006f8951204\" (UID: \"1e93adf3-6136-459e-89cf-0006f8951204\") " Oct 31 00:00:17.451126 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.450996 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e93adf3-6136-459e-89cf-0006f8951204-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e93adf3-6136-459e-89cf-0006f8951204-maas-api-service-ca\") pod \"1e93adf3-6136-459e-89cf-0006f8951204\" (UID: \"1e93adf3-6136-459e-89cf-0006f8951204\") " Oct 31 00:00:17.452581 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.452554 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e93adf3-6136-459e-89cf-0006f8951204-maas-api-service-ca" pod "1e93adf3-6136-459e-89cf-0006f8951204" (UID: "1e93adf3-6136-459e-89cf-0006f8951204"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:17.454254 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.454230 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e93adf3-6136-459e-89cf-0006f8951204-kube-api-access-49g56" pod "1e93adf3-6136-459e-89cf-0006f8951204" (UID: "1e93adf3-6136-459e-89cf-0006f8951204"). InnerVolumeSpecName "kube-api-access-49g56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:17.553619 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.553589 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49g56\" (UniqueName: \"kubernetes.io/projected/1e93adf3-6136-459e-89cf-0006f8951204-kube-api-access-49g56\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:17.553619 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:17.553614 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e93adf3-6136-459e-89cf-0006f8951204-maas-api-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:18.117018 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.116985 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp"] Oct 31 00:00:18.161964 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.161934 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" Oct 31 00:00:18.162120 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.161970 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z" event={"ID":"1e93adf3-6136-459e-89cf-0006f8951204","Type":"ContainerDied","Data":"12f51ca8cb7bb83eb6ce14fafd81d25441b9bac047e9ae8e2055864a23e6df6c"} Oct 31 00:00:18.162120 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.162025 7570 scope.go:122] "RemoveContainer" containerID="9c61ec523df5324bb5e3925237cd3bfb411eb96e3d2eb11c1fa6732256826d6e" Oct 31 00:00:18.164583 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.164559 7570 generic.go:356] "Generic (PLEG): container finished" podID="e4a67598-601d-4304-999f-243f467eb6db" containerID="a70279b6105b8ef22af7b00c7195d1164d05e8e289857c176820bb2e36dfe612" exitCode=60 Oct 31 00:00:18.164715 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.164617 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" event={"ID":"e4a67598-601d-4304-999f-243f467eb6db","Type":"ContainerDied","Data":"a70279b6105b8ef22af7b00c7195d1164d05e8e289857c176820bb2e36dfe612"} Oct 31 00:00:18.164933 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.164918 7570 scope.go:122] "RemoveContainer" containerID="a70279b6105b8ef22af7b00c7195d1164d05e8e289857c176820bb2e36dfe612" Oct 31 00:00:18.165155 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:18.165136 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch_odh-ai-gateway-infra(e4a67598-601d-4304-999f-243f467eb6db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" podUID="e4a67598-601d-4304-999f-243f467eb6db" Oct 31 00:00:18.174792 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.174769 7570 scope.go:122] "RemoveContainer" containerID="b170d3de492a86cd130c500f3cf7098ccce143f309ee7c6e90f89dac6780bb80" Oct 31 00:00:18.212421 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.212396 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch"] Oct 31 00:00:18.300681 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.297668 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh"] Oct 31 00:00:18.457635 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.457547 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq"] Oct 31 00:00:18.465250 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.465223 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" Oct 31 00:00:18.476721 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.476681 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z"] Oct 31 00:00:18.484097 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.484002 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-mvk6z"] Oct 31 00:00:18.632633 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.628812 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-maas-api-service-ca\") pod \"c42b94b3-7dc2-465f-9e1a-f128b015e5b8\" (UID: \"c42b94b3-7dc2-465f-9e1a-f128b015e5b8\") " Oct 31 00:00:18.632633 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.629386 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-kube-api-access-xz66p\" (UniqueName: \"kubernetes.io/projected/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-kube-api-access-xz66p\") pod \"c42b94b3-7dc2-465f-9e1a-f128b015e5b8\" (UID: \"c42b94b3-7dc2-465f-9e1a-f128b015e5b8\") " Oct 31 00:00:18.632633 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.632151 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-maas-api-service-ca" pod "c42b94b3-7dc2-465f-9e1a-f128b015e5b8" (UID: "c42b94b3-7dc2-465f-9e1a-f128b015e5b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:18.663516 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.663435 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-kube-api-access-xz66p" pod "c42b94b3-7dc2-465f-9e1a-f128b015e5b8" (UID: "c42b94b3-7dc2-465f-9e1a-f128b015e5b8"). InnerVolumeSpecName "kube-api-access-xz66p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:18.667740 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.667713 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e93adf3-6136-459e-89cf-0006f8951204" path="/var/lib/kubelet/pods/1e93adf3-6136-459e-89cf-0006f8951204/volumes" Oct 31 00:00:18.685713 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.679857 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk"] Oct 31 00:00:18.685713 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.684426 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4e6c3b-nr2rk"] Oct 31 00:00:18.706924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.706904 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" Oct 31 00:00:18.741964 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.741918 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:18.743441 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.743413 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:18.749839 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.749809 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b37651e-9dd9-4294-afa7-0bd3f427848c-kube-api-access-pxctl\" (UniqueName: \"kubernetes.io/projected/6b37651e-9dd9-4294-afa7-0bd3f427848c-kube-api-access-pxctl\") pod \"6b37651e-9dd9-4294-afa7-0bd3f427848c\" (UID: \"6b37651e-9dd9-4294-afa7-0bd3f427848c\") " Oct 31 00:00:18.750004 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.749980 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:00:18.750138 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.750103 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b37651e-9dd9-4294-afa7-0bd3f427848c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b37651e-9dd9-4294-afa7-0bd3f427848c-maas-api-service-ca\") pod \"6b37651e-9dd9-4294-afa7-0bd3f427848c\" (UID: \"6b37651e-9dd9-4294-afa7-0bd3f427848c\") " Oct 31 00:00:18.750868 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.750844 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-maas-api-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:18.750950 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.750875 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz66p\" (UniqueName: \"kubernetes.io/projected/c42b94b3-7dc2-465f-9e1a-f128b015e5b8-kube-api-access-xz66p\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:18.751216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.751190 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b37651e-9dd9-4294-afa7-0bd3f427848c-maas-api-service-ca" pod "6b37651e-9dd9-4294-afa7-0bd3f427848c" (UID: "6b37651e-9dd9-4294-afa7-0bd3f427848c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:18.756842 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.756793 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b37651e-9dd9-4294-afa7-0bd3f427848c-kube-api-access-pxctl" pod "6b37651e-9dd9-4294-afa7-0bd3f427848c" (UID: "6b37651e-9dd9-4294-afa7-0bd3f427848c"). InnerVolumeSpecName "kube-api-access-pxctl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:18.757643 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.757626 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" Oct 31 00:00:18.851796 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.851762 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02ffff16-0e62-47a6-9c29-4ad63ebbce79-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ffff16-0e62-47a6-9c29-4ad63ebbce79-maas-api-service-ca\") pod \"02ffff16-0e62-47a6-9c29-4ad63ebbce79\" (UID: \"02ffff16-0e62-47a6-9c29-4ad63ebbce79\") " Oct 31 00:00:18.851968 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.851825 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ffff16-0e62-47a6-9c29-4ad63ebbce79-kube-api-access-595cl\" (UniqueName: \"kubernetes.io/projected/02ffff16-0e62-47a6-9c29-4ad63ebbce79-kube-api-access-595cl\") pod \"02ffff16-0e62-47a6-9c29-4ad63ebbce79\" (UID: \"02ffff16-0e62-47a6-9c29-4ad63ebbce79\") " Oct 31 00:00:18.852223 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.852194 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ffff16-0e62-47a6-9c29-4ad63ebbce79-maas-api-service-ca" pod "02ffff16-0e62-47a6-9c29-4ad63ebbce79" (UID: "02ffff16-0e62-47a6-9c29-4ad63ebbce79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:18.852347 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.852329 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b37651e-9dd9-4294-afa7-0bd3f427848c-maas-api-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:18.852394 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.852355 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxctl\" (UniqueName: \"kubernetes.io/projected/6b37651e-9dd9-4294-afa7-0bd3f427848c-kube-api-access-pxctl\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:18.852394 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.852370 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ffff16-0e62-47a6-9c29-4ad63ebbce79-maas-api-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:18.854350 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.854284 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ffff16-0e62-47a6-9c29-4ad63ebbce79-kube-api-access-595cl" pod "02ffff16-0e62-47a6-9c29-4ad63ebbce79" (UID: "02ffff16-0e62-47a6-9c29-4ad63ebbce79"). InnerVolumeSpecName "kube-api-access-595cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:18.955218 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:18.955130 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-595cl\" (UniqueName: \"kubernetes.io/projected/02ffff16-0e62-47a6-9c29-4ad63ebbce79-kube-api-access-595cl\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:19.177016 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.176983 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" event={"ID":"6b37651e-9dd9-4294-afa7-0bd3f427848c","Type":"ContainerDied","Data":"e9ce7f94554e72d2a7fff9b4a66272ac575682b4acd69d644aa4cff4ec56d0f3"} Oct 31 00:00:19.177016 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.177002 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh" Oct 31 00:00:19.177530 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.177051 7570 scope.go:122] "RemoveContainer" containerID="fd74e0af3a0429e9e7f0fd1bc50dbe19a83b154409681f0a114297b2eb0dbdec" Oct 31 00:00:19.178396 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.178371 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" event={"ID":"c42b94b3-7dc2-465f-9e1a-f128b015e5b8","Type":"ContainerDied","Data":"61f96186595630bf4226fdfeff75b45632492d8bab07458d843a2c714fa431b0"} Oct 31 00:00:19.178530 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.178449 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp" Oct 31 00:00:19.181665 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.181633 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" Oct 31 00:00:19.181793 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.181666 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq" event={"ID":"02ffff16-0e62-47a6-9c29-4ad63ebbce79","Type":"ContainerDied","Data":"bea4101a792241e049699e6fd76c747b13f9dbbe19ea99f9b870097e55c2bc2e"} Oct 31 00:00:19.188023 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.188002 7570 scope.go:122] "RemoveContainer" containerID="fda1213b39a77702c54d081a4092b797b99e6abbceec0c1cb3dbf96595428705" Oct 31 00:00:19.199574 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.199558 7570 scope.go:122] "RemoveContainer" containerID="95edfc6b7f265f351fece984465b7c6b6f9412299bd9a08a33ab2f0cf93e06eb" Oct 31 00:00:19.215847 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.215820 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh"] Oct 31 00:00:19.233699 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.233673 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b567bb-29890080-m27hh"] Oct 31 00:00:19.262008 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.261984 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq"] Oct 31 00:00:19.266088 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.266065 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6b38a4-29890080-2drlq"] Oct 31 00:00:19.294043 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.294012 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp"] Oct 31 00:00:19.309155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.309127 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4e6c3b-29890080-pqdqp"] Oct 31 00:00:19.330525 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.330501 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" Oct 31 00:00:19.361708 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.361682 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a67598-601d-4304-999f-243f467eb6db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a67598-601d-4304-999f-243f467eb6db-maas-api-service-ca\") pod \"e4a67598-601d-4304-999f-243f467eb6db\" (UID: \"e4a67598-601d-4304-999f-243f467eb6db\") " Oct 31 00:00:19.361825 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.361809 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a67598-601d-4304-999f-243f467eb6db-kube-api-access-ws8gw\" (UniqueName: \"kubernetes.io/projected/e4a67598-601d-4304-999f-243f467eb6db-kube-api-access-ws8gw\") pod \"e4a67598-601d-4304-999f-243f467eb6db\" (UID: \"e4a67598-601d-4304-999f-243f467eb6db\") " Oct 31 00:00:19.362076 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.362055 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a67598-601d-4304-999f-243f467eb6db-maas-api-service-ca" pod "e4a67598-601d-4304-999f-243f467eb6db" (UID: "e4a67598-601d-4304-999f-243f467eb6db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:19.362399 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.362368 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a67598-601d-4304-999f-243f467eb6db-maas-api-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:19.364152 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.364126 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a67598-601d-4304-999f-243f467eb6db-kube-api-access-ws8gw" pod "e4a67598-601d-4304-999f-243f467eb6db" (UID: "e4a67598-601d-4304-999f-243f467eb6db"). InnerVolumeSpecName "kube-api-access-ws8gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:19.463693 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:19.463622 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8gw\" (UniqueName: \"kubernetes.io/projected/e4a67598-601d-4304-999f-243f467eb6db-kube-api-access-ws8gw\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:20.193552 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:20.193518 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" event={"ID":"e4a67598-601d-4304-999f-243f467eb6db","Type":"ContainerDied","Data":"597f9289adb57211c44a9a4cb3a6649ec3a1ee824593990baf7fe32046d2a918"} Oct 31 00:00:20.193552 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:20.193548 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch" Oct 31 00:00:20.194050 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:20.193589 7570 scope.go:122] "RemoveContainer" containerID="a70279b6105b8ef22af7b00c7195d1164d05e8e289857c176820bb2e36dfe612" Oct 31 00:00:20.230015 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:20.229986 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch"] Oct 31 00:00:20.236008 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:20.235986 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-885999-29890080-86pch"] Oct 31 00:00:20.605791 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:20.605754 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" path="/var/lib/kubelet/pods/02ffff16-0e62-47a6-9c29-4ad63ebbce79/volumes" Oct 31 00:00:20.606307 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:20.606290 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42fd7f0b-dfc4-4e90-a321-c4a51115daab" path="/var/lib/kubelet/pods/42fd7f0b-dfc4-4e90-a321-c4a51115daab/volumes" Oct 31 00:00:20.606686 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:20.606673 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" path="/var/lib/kubelet/pods/6b37651e-9dd9-4294-afa7-0bd3f427848c/volumes" Oct 31 00:00:20.607356 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:20.607341 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42b94b3-7dc2-465f-9e1a-f128b015e5b8" path="/var/lib/kubelet/pods/c42b94b3-7dc2-465f-9e1a-f128b015e5b8/volumes" Oct 31 00:00:20.607767 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:20.607754 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4a67598-601d-4304-999f-243f467eb6db" path="/var/lib/kubelet/pods/e4a67598-601d-4304-999f-243f467eb6db/volumes" Oct 31 00:00:23.134407 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:23.134371 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" podUID="7f016eec-7091-4a3e-847e-7a382085ffef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:00:23.177077 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:23.177051 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" podUID="c070c544-43c5-4796-b385-6c4094a63624" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:00:24.687577 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.687533 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn"] Oct 31 00:00:24.688063 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.687888 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" containerID="cri-o://a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571" gracePeriod=30 Oct 31 00:00:24.913403 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.913372 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h"] Oct 31 00:00:24.914170 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914148 7570 cpu_manager.go:414] "containerMap: removing container" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" containerName="cleanup" Oct 31 00:00:24.914170 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914169 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914183 7570 cpu_manager.go:414] "containerMap: removing container" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914188 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914195 7570 cpu_manager.go:414] "containerMap: removing container" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914201 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914212 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e4a67598-601d-4304-999f-243f467eb6db" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914216 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a67598-601d-4304-999f-243f467eb6db" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914224 7570 cpu_manager.go:414] "containerMap: removing container" podUID="1e93adf3-6136-459e-89cf-0006f8951204" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914228 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e93adf3-6136-459e-89cf-0006f8951204" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914236 7570 cpu_manager.go:414] "containerMap: removing container" podUID="1e93adf3-6136-459e-89cf-0006f8951204" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914241 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e93adf3-6136-459e-89cf-0006f8951204" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914248 7570 cpu_manager.go:414] "containerMap: removing container" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914252 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914259 7570 cpu_manager.go:414] "containerMap: removing container" podUID="c42b94b3-7dc2-465f-9e1a-f128b015e5b8" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914263 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42b94b3-7dc2-465f-9e1a-f128b015e5b8" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914269 7570 cpu_manager.go:414] "containerMap: removing container" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914274 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914286 7570 cpu_manager.go:414] "containerMap: removing container" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914290 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914295 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e4a67598-601d-4304-999f-243f467eb6db" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914300 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a67598-601d-4304-999f-243f467eb6db" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914307 7570 cpu_manager.go:414] "containerMap: removing container" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914313 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914318 7570 cpu_manager.go:414] "containerMap: removing container" podUID="1e93adf3-6136-459e-89cf-0006f8951204" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914322 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e93adf3-6136-459e-89cf-0006f8951204" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914330 7570 cpu_manager.go:414] "containerMap: removing container" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914334 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" containerName="cleanup" Oct 31 00:00:24.914325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914341 7570 cpu_manager.go:414] "containerMap: removing container" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914346 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914362 7570 cpu_manager.go:414] "containerMap: removing container" podUID="2ad2904e-63ba-45d4-b7dd-60f0edd67962" containerName="revoke-keys" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914368 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad2904e-63ba-45d4-b7dd-60f0edd67962" containerName="revoke-keys" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914484 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914496 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914503 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1bda07-310d-43a8-a926-9004487b0034" containerName="main" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914512 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a67598-601d-4304-999f-243f467eb6db" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914517 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e93adf3-6136-459e-89cf-0006f8951204" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914524 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e93adf3-6136-459e-89cf-0006f8951204" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914531 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42b94b3-7dc2-465f-9e1a-f128b015e5b8" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914536 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="80dc7b17-04ea-4d03-a843-24891d3cda07" containerName="main" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914543 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="d073cc7c-6856-4131-8bf7-250400f05d7f" containerName="main" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914548 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a67598-601d-4304-999f-243f467eb6db" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914553 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914559 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e93adf3-6136-459e-89cf-0006f8951204" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914564 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914572 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42b94b3-7dc2-465f-9e1a-f128b015e5b8" containerName="cleanup" Oct 31 00:00:24.915216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.914579 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad2904e-63ba-45d4-b7dd-60f0edd67962" containerName="revoke-keys" Oct 31 00:00:24.918201 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.918181 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:24.928397 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.928374 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h"] Oct 31 00:00:24.938623 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.938547 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn"] Oct 31 00:00:24.939504 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.939442 7570 cpu_manager.go:414] "containerMap: removing container" podUID="c42b94b3-7dc2-465f-9e1a-f128b015e5b8" containerName="cleanup" Oct 31 00:00:24.939504 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.939481 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42b94b3-7dc2-465f-9e1a-f128b015e5b8" containerName="cleanup" Oct 31 00:00:24.939646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.939519 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e4a67598-601d-4304-999f-243f467eb6db" containerName="cleanup" Oct 31 00:00:24.939646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.939528 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a67598-601d-4304-999f-243f467eb6db" containerName="cleanup" Oct 31 00:00:24.939754 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.939712 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b37651e-9dd9-4294-afa7-0bd3f427848c" containerName="cleanup" Oct 31 00:00:24.939754 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.939740 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ffff16-0e62-47a6-9c29-4ad63ebbce79" containerName="cleanup" Oct 31 00:00:24.939754 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.939753 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a67598-601d-4304-999f-243f467eb6db" containerName="cleanup" Oct 31 00:00:24.944726 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.944709 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:24.955996 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.955970 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn"] Oct 31 00:00:24.985725 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.985701 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:00:24.986549 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.986527 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:00:24.990766 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:24.990749 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 31 00:00:25.030857 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.030828 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2533ebe5-857f-4255-8c85-5a526fe1062b-kube-api-access-7ll6v\" (UniqueName: \"kubernetes.io/projected/2533ebe5-857f-4255-8c85-5a526fe1062b-kube-api-access-7ll6v\") pod \"2533ebe5-857f-4255-8c85-5a526fe1062b\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " Oct 31 00:00:25.030991 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.030975 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2533ebe5-857f-4255-8c85-5a526fe1062b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2533ebe5-857f-4255-8c85-5a526fe1062b-openshift-service-ca\") pod \"2533ebe5-857f-4255-8c85-5a526fe1062b\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " Oct 31 00:00:25.031087 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.031070 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-metrics-certs\") pod \"2533ebe5-857f-4255-8c85-5a526fe1062b\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " Oct 31 00:00:25.031196 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.031180 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-maas-api-tls\") pod \"2533ebe5-857f-4255-8c85-5a526fe1062b\" (UID: \"2533ebe5-857f-4255-8c85-5a526fe1062b\") " Oct 31 00:00:25.031611 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.031393 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2533ebe5-857f-4255-8c85-5a526fe1062b-openshift-service-ca" pod "2533ebe5-857f-4255-8c85-5a526fe1062b" (UID: "2533ebe5-857f-4255-8c85-5a526fe1062b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:25.031611 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.031500 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgdzk\" (UniqueName: \"kubernetes.io/projected/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-kube-api-access-wgdzk\") pod \"maas-api-revoke-sub-f0d0f229-4kzgn\" (UID: \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:25.031742 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.031650 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dglcb\" (UniqueName: \"kubernetes.io/projected/8b4f82dd-efe7-4cfc-a4b3-507773e99758-kube-api-access-dglcb\") pod \"maas-api-revoke-sub-aeee9160-4wv2h\" (UID: \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:25.031742 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.031728 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0d0f229-4kzgn\" (UID: \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:25.031742 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.031760 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0d0f229-4kzgn\" (UID: \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:25.032050 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.031855 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeee9160-4wv2h\" (UID: \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:25.032050 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.031900 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeee9160-4wv2h\" (UID: \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:25.032278 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.032255 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2533ebe5-857f-4255-8c85-5a526fe1062b-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:25.033245 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.033226 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2533ebe5-857f-4255-8c85-5a526fe1062b-kube-api-access-7ll6v" pod "2533ebe5-857f-4255-8c85-5a526fe1062b" (UID: "2533ebe5-857f-4255-8c85-5a526fe1062b"). InnerVolumeSpecName "kube-api-access-7ll6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:25.033934 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.033916 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-metrics-certs" pod "2533ebe5-857f-4255-8c85-5a526fe1062b" (UID: "2533ebe5-857f-4255-8c85-5a526fe1062b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:25.033997 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.033936 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-maas-api-tls" pod "2533ebe5-857f-4255-8c85-5a526fe1062b" (UID: "2533ebe5-857f-4255-8c85-5a526fe1062b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:25.133070 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.133044 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl"] Oct 31 00:00:25.133310 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.133289 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgdzk\" (UniqueName: \"kubernetes.io/projected/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-kube-api-access-wgdzk\") pod \"maas-api-revoke-sub-f0d0f229-4kzgn\" (UID: \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:25.133456 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.133434 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dglcb\" (UniqueName: \"kubernetes.io/projected/8b4f82dd-efe7-4cfc-a4b3-507773e99758-kube-api-access-dglcb\") pod \"maas-api-revoke-sub-aeee9160-4wv2h\" (UID: \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:25.133592 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.133573 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0d0f229-4kzgn\" (UID: \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:25.133661 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.133618 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0d0f229-4kzgn\" (UID: \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:25.133718 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.133657 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeee9160-4wv2h\" (UID: \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:25.133718 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.133689 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeee9160-4wv2h\" (UID: \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:25.133819 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.133810 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ll6v\" (UniqueName: \"kubernetes.io/projected/2533ebe5-857f-4255-8c85-5a526fe1062b-kube-api-access-7ll6v\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:25.133867 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.133825 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:25.133867 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.133834 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2533ebe5-857f-4255-8c85-5a526fe1062b-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:25.134336 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.134312 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0d0f229-4kzgn\" (UID: \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:25.134833 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.134781 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeee9160-4wv2h\" (UID: \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:25.136195 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.136173 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0d0f229-4kzgn\" (UID: \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:25.136315 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.136294 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeee9160-4wv2h\" (UID: \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:25.143224 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.143178 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-843d8c-qbtpl"] Oct 31 00:00:25.167495 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.167200 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dglcb\" (UniqueName: \"kubernetes.io/projected/8b4f82dd-efe7-4cfc-a4b3-507773e99758-kube-api-access-dglcb\") pod \"maas-api-revoke-sub-aeee9160-4wv2h\" (UID: \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:25.179532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.178872 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgdzk\" (UniqueName: \"kubernetes.io/projected/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-kube-api-access-wgdzk\") pod \"maas-api-revoke-sub-f0d0f229-4kzgn\" (UID: \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:25.243532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.242602 7570 generic.go:356] "Generic (PLEG): container finished" podID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerID="a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571" exitCode=0 Oct 31 00:00:25.243532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.242662 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" Oct 31 00:00:25.243532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.242683 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" event={"ID":"2533ebe5-857f-4255-8c85-5a526fe1062b","Type":"ContainerDied","Data":"a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571"} Oct 31 00:00:25.243532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.242739 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn" event={"ID":"2533ebe5-857f-4255-8c85-5a526fe1062b","Type":"ContainerDied","Data":"6273bf5bd11077cfe56efa75842ee377c1d94cd593224d7b771f06461a724cf9"} Oct 31 00:00:25.243532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.242766 7570 scope.go:122] "RemoveContainer" containerID="a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571" Oct 31 00:00:25.243932 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.243915 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:25.266795 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.266772 7570 scope.go:122] "RemoveContainer" containerID="3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076" Oct 31 00:00:25.279164 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.279133 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn"] Oct 31 00:00:25.283229 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.283206 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-843d8c-b5bc77847-nfscn"] Oct 31 00:00:25.287439 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.287421 7570 scope.go:122] "RemoveContainer" containerID="a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571" Oct 31 00:00:25.287869 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:25.287848 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571\": container with ID starting with a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571 not found: ID does not exist" containerID="a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571" Oct 31 00:00:25.287965 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.287883 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571"} err="failed to get container status \"a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571\": rpc error: code = NotFound desc = could not find container \"a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571\": container with ID starting with a66da4f7ceede89e5fbc8e9d541d85a5f3afb09d735137bae1870dcd5bf29571 not found: ID does not exist" Oct 31 00:00:25.287965 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.287906 7570 scope.go:122] "RemoveContainer" containerID="3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076" Oct 31 00:00:25.288283 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:25.288261 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076\": container with ID starting with 3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076 not found: ID does not exist" containerID="3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076" Oct 31 00:00:25.288345 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.288291 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076"} err="failed to get container status \"3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076\": rpc error: code = NotFound desc = could not find container \"3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076\": container with ID starting with 3742dd6c123b7a8e6e86a51a380539915cc0e082ea060246635618c51a27e076 not found: ID does not exist" Oct 31 00:00:25.289487 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.289454 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:25.557918 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.557895 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h"] Oct 31 00:00:25.558857 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:00:25.558822 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4f82dd_efe7_4cfc_a4b3_507773e99758.slice/crio-07e5889f94e2d9f0641833a349d1e189c7adabad162dc5d6320c007d60550c8c WatchSource:0}: Error finding container 07e5889f94e2d9f0641833a349d1e189c7adabad162dc5d6320c007d60550c8c: Status 404 returned error can't find the container with id 07e5889f94e2d9f0641833a349d1e189c7adabad162dc5d6320c007d60550c8c Oct 31 00:00:25.611916 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.611891 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn"] Oct 31 00:00:25.612214 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:00:25.612190 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94b6a9a_db8d_4e81_affe_c11f0e402d6b.slice/crio-993118a2c39e011a904cd646f620ca80982d25d0d4f1c565875ce3e86029c80f WatchSource:0}: Error finding container 993118a2c39e011a904cd646f620ca80982d25d0d4f1c565875ce3e86029c80f: Status 404 returned error can't find the container with id 993118a2c39e011a904cd646f620ca80982d25d0d4f1c565875ce3e86029c80f Oct 31 00:00:25.761599 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:25.761560 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" containerID="cri-o://47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f" gracePeriod=45 Oct 31 00:00:26.114157 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.114134 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 31 00:00:26.252719 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.252688 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-dshm\") pod \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " Oct 31 00:00:26.252880 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.252732 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-home\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-home\") pod \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " Oct 31 00:00:26.252880 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.252778 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed4e6002-31dc-4eb3-9988-d808eee76c7f-kube-api-access-wj4nc\" (UniqueName: \"kubernetes.io/projected/ed4e6002-31dc-4eb3-9988-d808eee76c7f-kube-api-access-wj4nc\") pod \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " Oct 31 00:00:26.252880 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.252810 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tls-certs\") pod \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " Oct 31 00:00:26.253050 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.252872 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" event={"ID":"e94b6a9a-db8d-4e81-affe-c11f0e402d6b","Type":"ContainerStarted","Data":"ff4818a396ddb4cb0ae97902881cf4ad80caa0c89f34ade12ce4ea534ba03c54"} Oct 31 00:00:26.253050 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.252935 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" event={"ID":"e94b6a9a-db8d-4e81-affe-c11f0e402d6b","Type":"ContainerStarted","Data":"993118a2c39e011a904cd646f620ca80982d25d0d4f1c565875ce3e86029c80f"} Oct 31 00:00:26.253050 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.252944 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tmp-dir\") pod \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " Oct 31 00:00:26.253182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.253059 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-home" pod "ed4e6002-31dc-4eb3-9988-d808eee76c7f" (UID: "ed4e6002-31dc-4eb3-9988-d808eee76c7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:26.253182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.253128 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-model-cache\") pod \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\" (UID: \"ed4e6002-31dc-4eb3-9988-d808eee76c7f\") " Oct 31 00:00:26.253380 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.253358 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tmp-dir" pod "ed4e6002-31dc-4eb3-9988-d808eee76c7f" (UID: "ed4e6002-31dc-4eb3-9988-d808eee76c7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:26.253755 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.253632 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-model-cache" pod "ed4e6002-31dc-4eb3-9988-d808eee76c7f" (UID: "ed4e6002-31dc-4eb3-9988-d808eee76c7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:26.254086 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.254036 7570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tmp-dir\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:26.254086 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.254064 7570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-model-cache\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:26.254086 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.254079 7570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-home\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:26.255913 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.255875 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tls-certs" pod "ed4e6002-31dc-4eb3-9988-d808eee76c7f" (UID: "ed4e6002-31dc-4eb3-9988-d808eee76c7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:26.256632 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.256605 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" event={"ID":"8b4f82dd-efe7-4cfc-a4b3-507773e99758","Type":"ContainerStarted","Data":"46b44bfd009290a9a959d8a63a0e60c9bd5890dd23edf0a8ca5204157b481c63"} Oct 31 00:00:26.256718 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.256658 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" event={"ID":"8b4f82dd-efe7-4cfc-a4b3-507773e99758","Type":"ContainerStarted","Data":"07e5889f94e2d9f0641833a349d1e189c7adabad162dc5d6320c007d60550c8c"} Oct 31 00:00:26.256820 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.256792 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4e6002-31dc-4eb3-9988-d808eee76c7f-kube-api-access-wj4nc" pod "ed4e6002-31dc-4eb3-9988-d808eee76c7f" (UID: "ed4e6002-31dc-4eb3-9988-d808eee76c7f"). InnerVolumeSpecName "kube-api-access-wj4nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:26.257819 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.257800 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-dshm" pod "ed4e6002-31dc-4eb3-9988-d808eee76c7f" (UID: "ed4e6002-31dc-4eb3-9988-d808eee76c7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:26.258314 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.258290 7570 generic.go:356] "Generic (PLEG): container finished" podID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerID="47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f" exitCode=0 Oct 31 00:00:26.258403 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.258324 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" Oct 31 00:00:26.258403 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.258371 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" event={"ID":"ed4e6002-31dc-4eb3-9988-d808eee76c7f","Type":"ContainerDied","Data":"47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f"} Oct 31 00:00:26.258564 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.258406 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f" event={"ID":"ed4e6002-31dc-4eb3-9988-d808eee76c7f","Type":"ContainerDied","Data":"be4405ad829b994ef067dcd8943fdf1c2e5174e4eb80110f94affedb0a1a9b59"} Oct 31 00:00:26.258564 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.258424 7570 scope.go:122] "RemoveContainer" containerID="47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f" Oct 31 00:00:26.281872 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.281851 7570 scope.go:122] "RemoveContainer" containerID="47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f" Oct 31 00:00:26.282162 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:26.282146 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f\": container with ID starting with 47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f not found: ID does not exist" containerID="47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f" Oct 31 00:00:26.282229 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.282171 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f"} err="failed to get container status \"47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f\": rpc error: code = NotFound desc = could not find container \"47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f\": container with ID starting with 47bef4cdf47b7504f5ddc208b6db202cca97c8200376ca23e4b2a1bb93ace91f not found: ID does not exist" Oct 31 00:00:26.289323 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.289281 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" podStartSLOduration=2.289269597 podStartE2EDuration="2.289269597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:26.286900597 +0000 UTC m=+2437.805901687" watchObservedRunningTime="2026-10-31 00:00:26.289269597 +0000 UTC m=+2437.808270686" Oct 31 00:00:26.312544 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.312489 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" podStartSLOduration=2.312475068 podStartE2EDuration="2.312475068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:26.30889225 +0000 UTC m=+2437.827893340" watchObservedRunningTime="2026-10-31 00:00:26.312475068 +0000 UTC m=+2437.831476160" Oct 31 00:00:26.325931 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.325879 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f"] Oct 31 00:00:26.340822 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.340794 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b567bb/premium-w3-b567bb-kserve-586f4c9b9-rn44f"] Oct 31 00:00:26.356100 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.356076 7570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed4e6002-31dc-4eb3-9988-d808eee76c7f-dshm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:26.356100 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.356101 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj4nc\" (UniqueName: \"kubernetes.io/projected/ed4e6002-31dc-4eb3-9988-d808eee76c7f-kube-api-access-wj4nc\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:26.356278 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.356111 7570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed4e6002-31dc-4eb3-9988-d808eee76c7f-tls-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:26.605931 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.605902 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" path="/var/lib/kubelet/pods/2533ebe5-857f-4255-8c85-5a526fe1062b/volumes" Oct 31 00:00:26.606914 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.606897 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4" path="/var/lib/kubelet/pods/56f9c3b3-0d17-47f2-a0db-7aa3f1bc2ec4/volumes" Oct 31 00:00:26.607279 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:26.607268 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" path="/var/lib/kubelet/pods/ed4e6002-31dc-4eb3-9988-d808eee76c7f/volumes" Oct 31 00:00:27.267069 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:27.267035 7570 generic.go:356] "Generic (PLEG): container finished" podID="e94b6a9a-db8d-4e81-affe-c11f0e402d6b" containerID="ff4818a396ddb4cb0ae97902881cf4ad80caa0c89f34ade12ce4ea534ba03c54" exitCode=0 Oct 31 00:00:27.267605 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:27.267124 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" event={"ID":"e94b6a9a-db8d-4e81-affe-c11f0e402d6b","Type":"ContainerDied","Data":"ff4818a396ddb4cb0ae97902881cf4ad80caa0c89f34ade12ce4ea534ba03c54"} Oct 31 00:00:27.269024 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:27.269002 7570 generic.go:356] "Generic (PLEG): container finished" podID="8b4f82dd-efe7-4cfc-a4b3-507773e99758" containerID="46b44bfd009290a9a959d8a63a0e60c9bd5890dd23edf0a8ca5204157b481c63" exitCode=0 Oct 31 00:00:27.269117 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:27.269031 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" event={"ID":"8b4f82dd-efe7-4cfc-a4b3-507773e99758","Type":"ContainerDied","Data":"46b44bfd009290a9a959d8a63a0e60c9bd5890dd23edf0a8ca5204157b481c63"} Oct 31 00:00:28.491896 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.491840 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:28.537551 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.537494 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:28.604260 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.604227 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-openshift-ca\") pod \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\" (UID: \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\") " Oct 31 00:00:28.604421 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.604276 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-xks-ca\") pod \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\" (UID: \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\") " Oct 31 00:00:28.604421 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.604358 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-kube-api-access-wgdzk\" (UniqueName: \"kubernetes.io/projected/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-kube-api-access-wgdzk\") pod \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\" (UID: \"e94b6a9a-db8d-4e81-affe-c11f0e402d6b\") " Oct 31 00:00:28.604421 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.604388 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-openshift-ca\") pod \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\" (UID: \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\") " Oct 31 00:00:28.604658 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.604630 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-openshift-ca" pod "e94b6a9a-db8d-4e81-affe-c11f0e402d6b" (UID: "e94b6a9a-db8d-4e81-affe-c11f0e402d6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:28.604713 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.604648 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-xks-ca\") pod \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\" (UID: \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\") " Oct 31 00:00:28.604778 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.604761 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4f82dd-efe7-4cfc-a4b3-507773e99758-kube-api-access-dglcb\" (UniqueName: \"kubernetes.io/projected/8b4f82dd-efe7-4cfc-a4b3-507773e99758-kube-api-access-dglcb\") pod \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\" (UID: \"8b4f82dd-efe7-4cfc-a4b3-507773e99758\") " Oct 31 00:00:28.605160 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.605135 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-openshift-ca" pod "8b4f82dd-efe7-4cfc-a4b3-507773e99758" (UID: "8b4f82dd-efe7-4cfc-a4b3-507773e99758"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:28.605581 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.605554 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:28.605581 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.605583 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:28.607656 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.607629 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-xks-ca" pod "8b4f82dd-efe7-4cfc-a4b3-507773e99758" (UID: "8b4f82dd-efe7-4cfc-a4b3-507773e99758"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:28.608033 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.608014 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4f82dd-efe7-4cfc-a4b3-507773e99758-kube-api-access-dglcb" pod "8b4f82dd-efe7-4cfc-a4b3-507773e99758" (UID: "8b4f82dd-efe7-4cfc-a4b3-507773e99758"). InnerVolumeSpecName "kube-api-access-dglcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:28.609733 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.609712 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-xks-ca" pod "e94b6a9a-db8d-4e81-affe-c11f0e402d6b" (UID: "e94b6a9a-db8d-4e81-affe-c11f0e402d6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:28.610605 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.610562 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-kube-api-access-wgdzk" pod "e94b6a9a-db8d-4e81-affe-c11f0e402d6b" (UID: "e94b6a9a-db8d-4e81-affe-c11f0e402d6b"). InnerVolumeSpecName "kube-api-access-wgdzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:28.706740 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.706713 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4f82dd-efe7-4cfc-a4b3-507773e99758-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:28.706740 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.706738 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dglcb\" (UniqueName: \"kubernetes.io/projected/8b4f82dd-efe7-4cfc-a4b3-507773e99758-kube-api-access-dglcb\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:28.706740 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.706750 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:28.707007 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:28.706763 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgdzk\" (UniqueName: \"kubernetes.io/projected/e94b6a9a-db8d-4e81-affe-c11f0e402d6b-kube-api-access-wgdzk\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:29.293203 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:29.293173 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" Oct 31 00:00:29.293203 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:29.293186 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn" event={"ID":"e94b6a9a-db8d-4e81-affe-c11f0e402d6b","Type":"ContainerDied","Data":"993118a2c39e011a904cd646f620ca80982d25d0d4f1c565875ce3e86029c80f"} Oct 31 00:00:29.293429 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:29.293242 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="993118a2c39e011a904cd646f620ca80982d25d0d4f1c565875ce3e86029c80f" Oct 31 00:00:29.294932 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:29.294914 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" Oct 31 00:00:29.295067 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:29.294955 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h" event={"ID":"8b4f82dd-efe7-4cfc-a4b3-507773e99758","Type":"ContainerDied","Data":"07e5889f94e2d9f0641833a349d1e189c7adabad162dc5d6320c007d60550c8c"} Oct 31 00:00:29.295067 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:29.294992 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07e5889f94e2d9f0641833a349d1e189c7adabad162dc5d6320c007d60550c8c" Oct 31 00:00:29.423652 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:29.421948 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:00:30.306021 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.305847 7570 generic.go:356] "Generic (PLEG): container finished" podID="c070c544-43c5-4796-b385-6c4094a63624" containerID="d2d0452178b2079a293c0673a10ebba578c88a88d0facf27ff846b2140edd577" exitCode=0 Oct 31 00:00:30.306021 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.305959 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" event={"ID":"c070c544-43c5-4796-b385-6c4094a63624","Type":"ContainerDied","Data":"d2d0452178b2079a293c0673a10ebba578c88a88d0facf27ff846b2140edd577"} Oct 31 00:00:30.307915 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.307887 7570 generic.go:356] "Generic (PLEG): container finished" podID="7f016eec-7091-4a3e-847e-7a382085ffef" containerID="65b340054ff204ea3e9089ea510d55bf4e3712e10df9dd29f7782872c7f0b071" exitCode=0 Oct 31 00:00:30.308038 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.307941 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" event={"ID":"7f016eec-7091-4a3e-847e-7a382085ffef","Type":"ContainerDied","Data":"65b340054ff204ea3e9089ea510d55bf4e3712e10df9dd29f7782872c7f0b071"} Oct 31 00:00:30.439010 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.438987 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 31 00:00:30.455677 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.455657 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 31 00:00:30.531082 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.531050 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f016eec-7091-4a3e-847e-7a382085ffef-kube-api-access-q6rz9\" (UniqueName: \"kubernetes.io/projected/7f016eec-7091-4a3e-847e-7a382085ffef-kube-api-access-q6rz9\") pod \"7f016eec-7091-4a3e-847e-7a382085ffef\" (UID: \"7f016eec-7091-4a3e-847e-7a382085ffef\") " Oct 31 00:00:30.531251 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.531232 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c070c544-43c5-4796-b385-6c4094a63624-kube-api-access-7k6hd\" (UniqueName: \"kubernetes.io/projected/c070c544-43c5-4796-b385-6c4094a63624-kube-api-access-7k6hd\") pod \"c070c544-43c5-4796-b385-6c4094a63624\" (UID: \"c070c544-43c5-4796-b385-6c4094a63624\") " Oct 31 00:00:30.531315 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.531277 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c070c544-43c5-4796-b385-6c4094a63624-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c070c544-43c5-4796-b385-6c4094a63624-plugins-config-volume\") pod \"c070c544-43c5-4796-b385-6c4094a63624\" (UID: \"c070c544-43c5-4796-b385-6c4094a63624\") " Oct 31 00:00:30.531409 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.531395 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f016eec-7091-4a3e-847e-7a382085ffef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f016eec-7091-4a3e-847e-7a382085ffef-plugins-config-volume\") pod \"7f016eec-7091-4a3e-847e-7a382085ffef\" (UID: \"7f016eec-7091-4a3e-847e-7a382085ffef\") " Oct 31 00:00:30.531761 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.531728 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c070c544-43c5-4796-b385-6c4094a63624-plugins-config-volume" pod "c070c544-43c5-4796-b385-6c4094a63624" (UID: "c070c544-43c5-4796-b385-6c4094a63624"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:30.531995 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.531968 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f016eec-7091-4a3e-847e-7a382085ffef-plugins-config-volume" pod "7f016eec-7091-4a3e-847e-7a382085ffef" (UID: "7f016eec-7091-4a3e-847e-7a382085ffef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:30.532298 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.532204 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c070c544-43c5-4796-b385-6c4094a63624-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:30.533779 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.533757 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c070c544-43c5-4796-b385-6c4094a63624-kube-api-access-7k6hd" pod "c070c544-43c5-4796-b385-6c4094a63624" (UID: "c070c544-43c5-4796-b385-6c4094a63624"). InnerVolumeSpecName "kube-api-access-7k6hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:30.533924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.533898 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f016eec-7091-4a3e-847e-7a382085ffef-kube-api-access-q6rz9" pod "7f016eec-7091-4a3e-847e-7a382085ffef" (UID: "7f016eec-7091-4a3e-847e-7a382085ffef"). InnerVolumeSpecName "kube-api-access-q6rz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:30.634096 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.634072 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f016eec-7091-4a3e-847e-7a382085ffef-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:30.634096 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.634098 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6rz9\" (UniqueName: \"kubernetes.io/projected/7f016eec-7091-4a3e-847e-7a382085ffef-kube-api-access-q6rz9\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:30.634218 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:30.634107 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k6hd\" (UniqueName: \"kubernetes.io/projected/c070c544-43c5-4796-b385-6c4094a63624-kube-api-access-7k6hd\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:31.316910 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:31.316845 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" Oct 31 00:00:31.317385 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:31.316840 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5" event={"ID":"c070c544-43c5-4796-b385-6c4094a63624","Type":"ContainerDied","Data":"08aec93d09964ae0c9f4b6d3936b47ab5a93421436c7d84e63dc60c04c83941d"} Oct 31 00:00:31.317385 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:31.317020 7570 scope.go:122] "RemoveContainer" containerID="d2d0452178b2079a293c0673a10ebba578c88a88d0facf27ff846b2140edd577" Oct 31 00:00:31.318540 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:31.318524 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" Oct 31 00:00:31.318640 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:31.318518 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg" event={"ID":"7f016eec-7091-4a3e-847e-7a382085ffef","Type":"ContainerDied","Data":"27b924632dc05efc69e605c5f5d276f45d0883a183b55b13381ce1f0c2108f12"} Oct 31 00:00:31.330081 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:31.330064 7570 scope.go:122] "RemoveContainer" containerID="65b340054ff204ea3e9089ea510d55bf4e3712e10df9dd29f7782872c7f0b071" Oct 31 00:00:31.351168 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:31.351146 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5"] Oct 31 00:00:31.355015 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:31.354995 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-843d8c-54fb6d9bbc-6dxj5"] Oct 31 00:00:31.371099 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:31.371078 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg"] Oct 31 00:00:31.378273 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:31.378255 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-843d8c-86bcd44599-hcpgg"] Oct 31 00:00:32.356001 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:32.355977 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 31 00:00:32.603096 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:32.603065 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f016eec-7091-4a3e-847e-7a382085ffef" path="/var/lib/kubelet/pods/7f016eec-7091-4a3e-847e-7a382085ffef/volumes" Oct 31 00:00:32.603497 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:32.603483 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c070c544-43c5-4796-b385-6c4094a63624" path="/var/lib/kubelet/pods/c070c544-43c5-4796-b385-6c4094a63624/volumes" Oct 31 00:00:35.052478 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:35.052427 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs"] Oct 31 00:00:35.059001 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:35.058973 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b567bb-9qtxs"] Oct 31 00:00:36.602422 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:36.602387 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a60236-f8f2-4188-b6a4-c74ceeb09143" path="/var/lib/kubelet/pods/26a60236-f8f2-4188-b6a4-c74ceeb09143/volumes" Oct 31 00:00:43.481697 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:43.481583 7570 scope.go:122] "RemoveContainer" containerID="23e7358568ccd8720464b8ec3cfa6637e23cbe8a66959e3052dd5bbddf4c7a4a" Oct 31 00:00:52.360510 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:52.360480 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 31 00:00:52.364342 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:52.364311 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5946dc94c7-djbgv" Oct 31 00:00:55.458043 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:55.458012 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd"] Oct 31 00:00:55.458428 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:55.458204 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" containerID="cri-o://ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5" gracePeriod=30 Oct 31 00:00:55.484303 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:55.484278 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp"] Oct 31 00:00:55.484500 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:55.484450 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" containerID="cri-o://9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b" gracePeriod=30 Oct 31 00:00:56.130323 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130289 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98"] Oct 31 00:00:56.130872 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130859 7570 cpu_manager.go:414] "containerMap: removing container" podUID="7f016eec-7091-4a3e-847e-7a382085ffef" containerName="payload-processing" Oct 31 00:00:56.130921 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130873 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f016eec-7091-4a3e-847e-7a382085ffef" containerName="payload-processing" Oct 31 00:00:56.130921 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130892 7570 cpu_manager.go:414] "containerMap: removing container" podUID="c070c544-43c5-4796-b385-6c4094a63624" containerName="payload-pre-processing" Oct 31 00:00:56.130921 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130897 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c070c544-43c5-4796-b385-6c4094a63624" containerName="payload-pre-processing" Oct 31 00:00:56.130921 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130916 7570 cpu_manager.go:414] "containerMap: removing container" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" Oct 31 00:00:56.130921 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130921 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" Oct 31 00:00:56.131067 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130926 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e94b6a9a-db8d-4e81-affe-c11f0e402d6b" containerName="revoke-keys" Oct 31 00:00:56.131067 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130931 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94b6a9a-db8d-4e81-affe-c11f0e402d6b" containerName="revoke-keys" Oct 31 00:00:56.131067 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130955 7570 cpu_manager.go:414] "containerMap: removing container" podUID="8b4f82dd-efe7-4cfc-a4b3-507773e99758" containerName="revoke-keys" Oct 31 00:00:56.131067 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130959 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4f82dd-efe7-4cfc-a4b3-507773e99758" containerName="revoke-keys" Oct 31 00:00:56.131067 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130967 7570 cpu_manager.go:414] "containerMap: removing container" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" Oct 31 00:00:56.131067 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.130971 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" Oct 31 00:00:56.131067 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.131052 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f016eec-7091-4a3e-847e-7a382085ffef" containerName="payload-processing" Oct 31 00:00:56.131067 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.131062 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c070c544-43c5-4796-b385-6c4094a63624" containerName="payload-pre-processing" Oct 31 00:00:56.131277 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.131072 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" Oct 31 00:00:56.131277 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.131078 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94b6a9a-db8d-4e81-affe-c11f0e402d6b" containerName="revoke-keys" Oct 31 00:00:56.131277 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.131088 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4f82dd-efe7-4cfc-a4b3-507773e99758" containerName="revoke-keys" Oct 31 00:00:56.131277 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.131093 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" Oct 31 00:00:56.131277 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.131101 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed4e6002-31dc-4eb3-9988-d808eee76c7f" containerName="main" Oct 31 00:00:56.134584 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.134569 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:00:56.151581 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.151560 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98"] Oct 31 00:00:56.174145 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.174124 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:00:56.174739 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.174717 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:00:56.264315 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.264287 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98\" (UID: \"b4414480-554a-4a1b-b70a-49e75e243696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:00:56.264517 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.264489 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98\" (UID: \"b4414480-554a-4a1b-b70a-49e75e243696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:00:56.264577 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.264527 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5wn\" (UniqueName: \"kubernetes.io/projected/b4414480-554a-4a1b-b70a-49e75e243696-kube-api-access-lf5wn\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98\" (UID: \"b4414480-554a-4a1b-b70a-49e75e243696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:00:56.365622 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.365598 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98\" (UID: \"b4414480-554a-4a1b-b70a-49e75e243696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:00:56.365728 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.365631 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5wn\" (UniqueName: \"kubernetes.io/projected/b4414480-554a-4a1b-b70a-49e75e243696-kube-api-access-lf5wn\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98\" (UID: \"b4414480-554a-4a1b-b70a-49e75e243696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:00:56.365787 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.365749 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98\" (UID: \"b4414480-554a-4a1b-b70a-49e75e243696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:00:56.366238 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.366217 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98\" (UID: \"b4414480-554a-4a1b-b70a-49e75e243696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:00:56.368573 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.368546 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98\" (UID: \"b4414480-554a-4a1b-b70a-49e75e243696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:00:56.385864 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.385814 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5wn\" (UniqueName: \"kubernetes.io/projected/b4414480-554a-4a1b-b70a-49e75e243696-kube-api-access-lf5wn\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98\" (UID: \"b4414480-554a-4a1b-b70a-49e75e243696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:00:56.448880 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.448859 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:00:56.733281 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:56.733257 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98"] Oct 31 00:00:56.733900 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:00:56.733878 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4414480_554a_4a1b_b70a_49e75e243696.slice/crio-71288c7f5da6e64cdc453301fbc8f317ab14aad55fc42a11b0b67ad74f871e60 WatchSource:0}: Error finding container 71288c7f5da6e64cdc453301fbc8f317ab14aad55fc42a11b0b67ad74f871e60: Status 404 returned error can't find the container with id 71288c7f5da6e64cdc453301fbc8f317ab14aad55fc42a11b0b67ad74f871e60 Oct 31 00:00:57.539173 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:57.539134 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" event={"ID":"b4414480-554a-4a1b-b70a-49e75e243696","Type":"ContainerStarted","Data":"f74018888a3f838d1480197109c8cb52c10f5ef0833e4cb163dce688a93659e1"} Oct 31 00:00:57.539416 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:57.539195 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" event={"ID":"b4414480-554a-4a1b-b70a-49e75e243696","Type":"ContainerStarted","Data":"71288c7f5da6e64cdc453301fbc8f317ab14aad55fc42a11b0b67ad74f871e60"} Oct 31 00:00:57.563986 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:57.563931 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" podStartSLOduration=1.563915135 podStartE2EDuration="1.563915135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:57.561621332 +0000 UTC m=+2469.080622422" watchObservedRunningTime="2026-10-31 00:00:57.563915135 +0000 UTC m=+2469.082916225" Oct 31 00:00:58.142410 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:58.142374 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:00:58.142803 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:58.142431 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:00:58.143947 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:58.143917 7570 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:00:58.144063 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:58.143973 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:00:58.548201 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:58.548120 7570 generic.go:356] "Generic (PLEG): container finished" podID="b4414480-554a-4a1b-b70a-49e75e243696" containerID="f74018888a3f838d1480197109c8cb52c10f5ef0833e4cb163dce688a93659e1" exitCode=0 Oct 31 00:00:58.548337 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:58.548201 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" event={"ID":"b4414480-554a-4a1b-b70a-49e75e243696","Type":"ContainerDied","Data":"f74018888a3f838d1480197109c8cb52c10f5ef0833e4cb163dce688a93659e1"} Oct 31 00:00:59.412148 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:00:59.412112 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:00:59.701968 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:59.701945 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:00:59.779419 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:59.779372 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd"] Oct 31 00:00:59.779660 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:59.779638 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" podUID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" containerName="istio-proxy" containerID="cri-o://13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21" gracePeriod=30 Oct 31 00:00:59.814606 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:59.814581 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-xks-ca\") pod \"b4414480-554a-4a1b-b70a-49e75e243696\" (UID: \"b4414480-554a-4a1b-b70a-49e75e243696\") " Oct 31 00:00:59.814720 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:59.814692 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-openshift-ca\") pod \"b4414480-554a-4a1b-b70a-49e75e243696\" (UID: \"b4414480-554a-4a1b-b70a-49e75e243696\") " Oct 31 00:00:59.814921 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:59.814905 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4414480-554a-4a1b-b70a-49e75e243696-kube-api-access-lf5wn\" (UniqueName: \"kubernetes.io/projected/b4414480-554a-4a1b-b70a-49e75e243696-kube-api-access-lf5wn\") pod \"b4414480-554a-4a1b-b70a-49e75e243696\" (UID: \"b4414480-554a-4a1b-b70a-49e75e243696\") " Oct 31 00:00:59.815098 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:59.815068 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-openshift-ca" pod "b4414480-554a-4a1b-b70a-49e75e243696" (UID: "b4414480-554a-4a1b-b70a-49e75e243696"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:59.815422 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:59.815406 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:59.816825 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:59.816802 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-xks-ca" pod "b4414480-554a-4a1b-b70a-49e75e243696" (UID: "b4414480-554a-4a1b-b70a-49e75e243696"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:59.816989 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:59.816970 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4414480-554a-4a1b-b70a-49e75e243696-kube-api-access-lf5wn" pod "b4414480-554a-4a1b-b70a-49e75e243696" (UID: "b4414480-554a-4a1b-b70a-49e75e243696"). InnerVolumeSpecName "kube-api-access-lf5wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:59.916921 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:59.916883 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4414480-554a-4a1b-b70a-49e75e243696-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:59.916921 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:00:59.916921 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf5wn\" (UniqueName: \"kubernetes.io/projected/b4414480-554a-4a1b-b70a-49e75e243696-kube-api-access-lf5wn\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:00.283785 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.283754 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t"] Oct 31 00:01:00.284323 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.284310 7570 cpu_manager.go:414] "containerMap: removing container" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" Oct 31 00:01:00.284372 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.284324 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2533ebe5-857f-4255-8c85-5a526fe1062b" containerName="maas-api" Oct 31 00:01:00.284372 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.284344 7570 cpu_manager.go:414] "containerMap: removing container" podUID="b4414480-554a-4a1b-b70a-49e75e243696" containerName="revoke-keys" Oct 31 00:01:00.284372 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.284350 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4414480-554a-4a1b-b70a-49e75e243696" containerName="revoke-keys" Oct 31 00:01:00.284512 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.284476 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4414480-554a-4a1b-b70a-49e75e243696" containerName="revoke-keys" Oct 31 00:01:00.287917 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.287901 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:00.299308 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.299288 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t"] Oct 31 00:01:00.422800 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.422773 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvhgw\" (UniqueName: \"kubernetes.io/projected/6f307a0d-2b1a-48ab-893c-7d983abab906-kube-api-access-qvhgw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t\" (UID: \"6f307a0d-2b1a-48ab-893c-7d983abab906\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:00.422800 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.422803 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t\" (UID: \"6f307a0d-2b1a-48ab-893c-7d983abab906\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:00.423190 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.422980 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t\" (UID: \"6f307a0d-2b1a-48ab-893c-7d983abab906\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:00.524143 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.524117 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvhgw\" (UniqueName: \"kubernetes.io/projected/6f307a0d-2b1a-48ab-893c-7d983abab906-kube-api-access-qvhgw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t\" (UID: \"6f307a0d-2b1a-48ab-893c-7d983abab906\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:00.524238 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.524150 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t\" (UID: \"6f307a0d-2b1a-48ab-893c-7d983abab906\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:00.524303 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.524289 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t\" (UID: \"6f307a0d-2b1a-48ab-893c-7d983abab906\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:00.524899 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.524871 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t\" (UID: \"6f307a0d-2b1a-48ab-893c-7d983abab906\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:00.526766 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.526746 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t\" (UID: \"6f307a0d-2b1a-48ab-893c-7d983abab906\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:00.536789 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.536770 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvhgw\" (UniqueName: \"kubernetes.io/projected/6f307a0d-2b1a-48ab-893c-7d983abab906-kube-api-access-qvhgw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t\" (UID: \"6f307a0d-2b1a-48ab-893c-7d983abab906\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:00.563268 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.563245 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" event={"ID":"b4414480-554a-4a1b-b70a-49e75e243696","Type":"ContainerDied","Data":"71288c7f5da6e64cdc453301fbc8f317ab14aad55fc42a11b0b67ad74f871e60"} Oct 31 00:01:00.563268 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.563265 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98" Oct 31 00:01:00.563417 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.563295 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71288c7f5da6e64cdc453301fbc8f317ab14aad55fc42a11b0b67ad74f871e60" Oct 31 00:01:00.603503 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.603440 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:00.819887 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.819866 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 31 00:01:00.823575 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.823555 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 31 00:01:00.902158 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.902130 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t"] Oct 31 00:01:00.904881 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:01:00.904844 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f307a0d_2b1a_48ab_893c_7d983abab906.slice/crio-2647cd0a4cbb73e11008aa8fd0a2e885bf5ddcedc77f9a4cd2c0a731e07f5577 WatchSource:0}: Error finding container 2647cd0a4cbb73e11008aa8fd0a2e885bf5ddcedc77f9a4cd2c0a731e07f5577: Status 404 returned error can't find the container with id 2647cd0a4cbb73e11008aa8fd0a2e885bf5ddcedc77f9a4cd2c0a731e07f5577 Oct 31 00:01:00.929878 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.929860 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-kube-api-access-xl6rc\" (UniqueName: \"kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-kube-api-access-xl6rc\") pod \"a74b8dd0-7067-44fd-94c0-948568ca285c\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " Oct 31 00:01:00.929965 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.929887 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-kube-api-access-m4zm9\" (UniqueName: \"kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-kube-api-access-m4zm9\") pod \"61c25f85-fbde-4c00-94c9-502637e57ba5\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " Oct 31 00:01:00.929965 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.929941 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-certs\") pod \"a74b8dd0-7067-44fd-94c0-948568ca285c\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " Oct 31 00:01:00.930040 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.929965 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-envoy\") pod \"a74b8dd0-7067-44fd-94c0-948568ca285c\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " Oct 31 00:01:00.930040 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930003 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-certs\") pod \"61c25f85-fbde-4c00-94c9-502637e57ba5\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " Oct 31 00:01:00.930138 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930101 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-data\") pod \"61c25f85-fbde-4c00-94c9-502637e57ba5\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " Oct 31 00:01:00.930216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930195 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-socket\") pod \"61c25f85-fbde-4c00-94c9-502637e57ba5\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " Oct 31 00:01:00.930270 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930237 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-socket\") pod \"a74b8dd0-7067-44fd-94c0-948568ca285c\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " Oct 31 00:01:00.930321 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930291 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-envoy\") pod \"61c25f85-fbde-4c00-94c9-502637e57ba5\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " Oct 31 00:01:00.930372 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930292 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-certs" pod "a74b8dd0-7067-44fd-94c0-948568ca285c" (UID: "a74b8dd0-7067-44fd-94c0-948568ca285c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:00.930372 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930324 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-certs" pod "61c25f85-fbde-4c00-94c9-502637e57ba5" (UID: "61c25f85-fbde-4c00-94c9-502637e57ba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:00.930372 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930320 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-podinfo\") pod \"61c25f85-fbde-4c00-94c9-502637e57ba5\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " Oct 31 00:01:00.930548 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930365 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-data" pod "61c25f85-fbde-4c00-94c9-502637e57ba5" (UID: "61c25f85-fbde-4c00-94c9-502637e57ba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:00.930548 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930408 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-podinfo\") pod \"a74b8dd0-7067-44fd-94c0-948568ca285c\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " Oct 31 00:01:00.930548 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930445 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-credential-socket\") pod \"a74b8dd0-7067-44fd-94c0-948568ca285c\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " Oct 31 00:01:00.930548 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930533 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-credential-socket\") pod \"61c25f85-fbde-4c00-94c9-502637e57ba5\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " Oct 31 00:01:00.930806 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930551 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-socket" pod "a74b8dd0-7067-44fd-94c0-948568ca285c" (UID: "a74b8dd0-7067-44fd-94c0-948568ca285c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:00.930806 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930564 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74b8dd0-7067-44fd-94c0-948568ca285c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74b8dd0-7067-44fd-94c0-948568ca285c-istiod-ca-cert\") pod \"a74b8dd0-7067-44fd-94c0-948568ca285c\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " Oct 31 00:01:00.930806 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930587 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-data\") pod \"a74b8dd0-7067-44fd-94c0-948568ca285c\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " Oct 31 00:01:00.930806 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930610 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-token\" (UniqueName: \"kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-token\") pod \"a74b8dd0-7067-44fd-94c0-948568ca285c\" (UID: \"a74b8dd0-7067-44fd-94c0-948568ca285c\") " Oct 31 00:01:00.930806 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930695 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c25f85-fbde-4c00-94c9-502637e57ba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c25f85-fbde-4c00-94c9-502637e57ba5-istiod-ca-cert\") pod \"61c25f85-fbde-4c00-94c9-502637e57ba5\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " Oct 31 00:01:00.930806 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930733 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-token\" (UniqueName: \"kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-token\") pod \"61c25f85-fbde-4c00-94c9-502637e57ba5\" (UID: \"61c25f85-fbde-4c00-94c9-502637e57ba5\") " Oct 31 00:01:00.930806 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.930747 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-socket" pod "61c25f85-fbde-4c00-94c9-502637e57ba5" (UID: "61c25f85-fbde-4c00-94c9-502637e57ba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:00.931128 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931030 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74b8dd0-7067-44fd-94c0-948568ca285c-istiod-ca-cert" pod "a74b8dd0-7067-44fd-94c0-948568ca285c" (UID: "a74b8dd0-7067-44fd-94c0-948568ca285c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:00.931343 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931188 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-data" pod "a74b8dd0-7067-44fd-94c0-948568ca285c" (UID: "a74b8dd0-7067-44fd-94c0-948568ca285c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:00.931343 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931305 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c25f85-fbde-4c00-94c9-502637e57ba5-istiod-ca-cert" pod "61c25f85-fbde-4c00-94c9-502637e57ba5" (UID: "61c25f85-fbde-4c00-94c9-502637e57ba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:00.931627 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931452 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-credential-socket" pod "61c25f85-fbde-4c00-94c9-502637e57ba5" (UID: "61c25f85-fbde-4c00-94c9-502637e57ba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:00.931692 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931627 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c25f85-fbde-4c00-94c9-502637e57ba5-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:00.931692 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931667 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:00.931904 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931721 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:00.931904 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931739 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:00.931904 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931743 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-credential-socket" pod "a74b8dd0-7067-44fd-94c0-948568ca285c" (UID: "a74b8dd0-7067-44fd-94c0-948568ca285c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:00.931904 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931755 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:00.931904 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931770 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:00.931904 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931783 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74b8dd0-7067-44fd-94c0-948568ca285c-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:00.931904 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.931796 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:00.934399 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.934286 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-envoy" pod "61c25f85-fbde-4c00-94c9-502637e57ba5" (UID: "61c25f85-fbde-4c00-94c9-502637e57ba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:00.934399 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.934334 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-envoy" pod "a74b8dd0-7067-44fd-94c0-948568ca285c" (UID: "a74b8dd0-7067-44fd-94c0-948568ca285c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:00.934399 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.934338 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-podinfo" pod "a74b8dd0-7067-44fd-94c0-948568ca285c" (UID: "a74b8dd0-7067-44fd-94c0-948568ca285c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:01:00.934937 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.934914 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-token" pod "61c25f85-fbde-4c00-94c9-502637e57ba5" (UID: "61c25f85-fbde-4c00-94c9-502637e57ba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:00.934937 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.934927 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-podinfo" pod "61c25f85-fbde-4c00-94c9-502637e57ba5" (UID: "61c25f85-fbde-4c00-94c9-502637e57ba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:01:00.935052 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.934939 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-kube-api-access-m4zm9" pod "61c25f85-fbde-4c00-94c9-502637e57ba5" (UID: "61c25f85-fbde-4c00-94c9-502637e57ba5"). InnerVolumeSpecName "kube-api-access-m4zm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:00.935052 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.934937 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-kube-api-access-xl6rc" pod "a74b8dd0-7067-44fd-94c0-948568ca285c" (UID: "a74b8dd0-7067-44fd-94c0-948568ca285c"). InnerVolumeSpecName "kube-api-access-xl6rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:00.935052 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:00.934949 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-token" pod "a74b8dd0-7067-44fd-94c0-948568ca285c" (UID: "a74b8dd0-7067-44fd-94c0-948568ca285c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:01.033215 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.033196 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:01.033299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.033217 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:01.033299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.033227 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6rc\" (UniqueName: \"kubernetes.io/projected/a74b8dd0-7067-44fd-94c0-948568ca285c-kube-api-access-xl6rc\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:01.033299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.033241 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4zm9\" (UniqueName: \"kubernetes.io/projected/61c25f85-fbde-4c00-94c9-502637e57ba5-kube-api-access-m4zm9\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:01.033299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.033253 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:01.033299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.033261 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:01.033299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.033270 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c25f85-fbde-4c00-94c9-502637e57ba5-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:01.033299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.033279 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74b8dd0-7067-44fd-94c0-948568ca285c-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:01.033299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.033287 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74b8dd0-7067-44fd-94c0-948568ca285c-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:01.033299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.033295 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c25f85-fbde-4c00-94c9-502637e57ba5-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:01.572033 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.571950 7570 generic.go:356] "Generic (PLEG): container finished" podID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerID="9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b" exitCode=0 Oct 31 00:01:01.572033 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.571998 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" Oct 31 00:01:01.572615 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.572042 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" event={"ID":"61c25f85-fbde-4c00-94c9-502637e57ba5","Type":"ContainerDied","Data":"9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b"} Oct 31 00:01:01.572615 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.572103 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp" event={"ID":"61c25f85-fbde-4c00-94c9-502637e57ba5","Type":"ContainerDied","Data":"61ddd0abdafe7f5014952ede2949d96149fe2fbd16e4bad29537b87f0d8b2a71"} Oct 31 00:01:01.572615 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.572135 7570 scope.go:122] "RemoveContainer" containerID="9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b" Oct 31 00:01:01.573619 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.573595 7570 generic.go:356] "Generic (PLEG): container finished" podID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerID="ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5" exitCode=0 Oct 31 00:01:01.573740 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.573655 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" Oct 31 00:01:01.573740 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.573670 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" event={"ID":"a74b8dd0-7067-44fd-94c0-948568ca285c","Type":"ContainerDied","Data":"ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5"} Oct 31 00:01:01.573740 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.573704 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd" event={"ID":"a74b8dd0-7067-44fd-94c0-948568ca285c","Type":"ContainerDied","Data":"a05def0e1f635f4aa81ef11eb9893b4ec5f7341b60b7943c9f780a81bf734901"} Oct 31 00:01:01.575520 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.575495 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" event={"ID":"6f307a0d-2b1a-48ab-893c-7d983abab906","Type":"ContainerStarted","Data":"218239c48ae3a7889de183724fc116f117419a6339706ca8313b4216a048c310"} Oct 31 00:01:01.575641 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.575542 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" event={"ID":"6f307a0d-2b1a-48ab-893c-7d983abab906","Type":"ContainerStarted","Data":"2647cd0a4cbb73e11008aa8fd0a2e885bf5ddcedc77f9a4cd2c0a731e07f5577"} Oct 31 00:01:01.584404 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.584385 7570 scope.go:122] "RemoveContainer" containerID="9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b" Oct 31 00:01:01.584919 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:01.584667 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b\": container with ID starting with 9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b not found: ID does not exist" containerID="9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b" Oct 31 00:01:01.584919 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.584696 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b"} err="failed to get container status \"9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b\": rpc error: code = NotFound desc = could not find container \"9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b\": container with ID starting with 9733466a165fe5793f87399e7b68782b08373fa7105646dd9896e2be5700ac9b not found: ID does not exist" Oct 31 00:01:01.584919 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.584719 7570 scope.go:122] "RemoveContainer" containerID="ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5" Oct 31 00:01:01.595161 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.595118 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" podStartSLOduration=1.595105284 podStartE2EDuration="1.595105284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:01.594337467 +0000 UTC m=+2473.113338556" watchObservedRunningTime="2026-10-31 00:01:01.595105284 +0000 UTC m=+2473.114106371" Oct 31 00:01:01.597114 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.596950 7570 scope.go:122] "RemoveContainer" containerID="ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5" Oct 31 00:01:01.597229 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:01.597212 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5\": container with ID starting with ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5 not found: ID does not exist" containerID="ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5" Oct 31 00:01:01.597268 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.597234 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5"} err="failed to get container status \"ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5\": rpc error: code = NotFound desc = could not find container \"ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5\": container with ID starting with ea7d2d17b46e476830d062b6688f3f2693a5a840d0fe9b1b7dc2ef8e44f3fcb5 not found: ID does not exist" Oct 31 00:01:01.615027 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.615000 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd"] Oct 31 00:01:01.619767 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.619746 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-p4hgd"] Oct 31 00:01:01.632384 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.632331 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp"] Oct 31 00:01:01.635942 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:01.635920 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-n4wbp"] Oct 31 00:01:02.583972 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:02.583934 7570 generic.go:356] "Generic (PLEG): container finished" podID="6f307a0d-2b1a-48ab-893c-7d983abab906" containerID="218239c48ae3a7889de183724fc116f117419a6339706ca8313b4216a048c310" exitCode=0 Oct 31 00:01:02.584368 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:02.584026 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" event={"ID":"6f307a0d-2b1a-48ab-893c-7d983abab906","Type":"ContainerDied","Data":"218239c48ae3a7889de183724fc116f117419a6339706ca8313b4216a048c310"} Oct 31 00:01:02.607599 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:02.607562 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" path="/var/lib/kubelet/pods/61c25f85-fbde-4c00-94c9-502637e57ba5/volumes" Oct 31 00:01:02.608645 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:02.608622 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" path="/var/lib/kubelet/pods/a74b8dd0-7067-44fd-94c0-948568ca285c/volumes" Oct 31 00:01:03.745259 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:03.745229 7570 patch_prober.go:29] interesting pod/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:01:03.745629 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:03.745287 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" podUID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:01:03.754345 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:03.754324 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:03.879063 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:03.878996 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-openshift-ca\") pod \"6f307a0d-2b1a-48ab-893c-7d983abab906\" (UID: \"6f307a0d-2b1a-48ab-893c-7d983abab906\") " Oct 31 00:01:03.879174 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:03.879101 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-xks-ca\") pod \"6f307a0d-2b1a-48ab-893c-7d983abab906\" (UID: \"6f307a0d-2b1a-48ab-893c-7d983abab906\") " Oct 31 00:01:03.879174 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:03.879125 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f307a0d-2b1a-48ab-893c-7d983abab906-kube-api-access-qvhgw\" (UniqueName: \"kubernetes.io/projected/6f307a0d-2b1a-48ab-893c-7d983abab906-kube-api-access-qvhgw\") pod \"6f307a0d-2b1a-48ab-893c-7d983abab906\" (UID: \"6f307a0d-2b1a-48ab-893c-7d983abab906\") " Oct 31 00:01:03.879326 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:03.879306 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-openshift-ca" pod "6f307a0d-2b1a-48ab-893c-7d983abab906" (UID: "6f307a0d-2b1a-48ab-893c-7d983abab906"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:03.879620 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:03.879594 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:03.881499 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:03.881480 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-xks-ca" pod "6f307a0d-2b1a-48ab-893c-7d983abab906" (UID: "6f307a0d-2b1a-48ab-893c-7d983abab906"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:03.881499 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:03.881455 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f307a0d-2b1a-48ab-893c-7d983abab906-kube-api-access-qvhgw" pod "6f307a0d-2b1a-48ab-893c-7d983abab906" (UID: "6f307a0d-2b1a-48ab-893c-7d983abab906"). InnerVolumeSpecName "kube-api-access-qvhgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:03.980979 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:03.980955 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f307a0d-2b1a-48ab-893c-7d983abab906-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:03.980979 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:03.980979 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvhgw\" (UniqueName: \"kubernetes.io/projected/6f307a0d-2b1a-48ab-893c-7d983abab906-kube-api-access-qvhgw\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:04.601950 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:04.601920 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" Oct 31 00:01:04.603385 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:04.603347 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t" event={"ID":"6f307a0d-2b1a-48ab-893c-7d983abab906","Type":"ContainerDied","Data":"2647cd0a4cbb73e11008aa8fd0a2e885bf5ddcedc77f9a4cd2c0a731e07f5577"} Oct 31 00:01:04.603385 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:04.603390 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2647cd0a4cbb73e11008aa8fd0a2e885bf5ddcedc77f9a4cd2c0a731e07f5577" Oct 31 00:01:05.051315 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.051289 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:01:05.196784 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.196708 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-socket\") pod \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " Oct 31 00:01:05.196784 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.196771 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-data\") pod \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " Oct 31 00:01:05.196991 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.196799 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-credential-socket\") pod \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " Oct 31 00:01:05.196991 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.196833 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istiod-ca-cert\") pod \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " Oct 31 00:01:05.196991 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.196874 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-token\" (UniqueName: \"kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-token\") pod \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " Oct 31 00:01:05.196991 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.196904 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-kube-api-access-5bkxr\" (UniqueName: \"kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-kube-api-access-5bkxr\") pod \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " Oct 31 00:01:05.196991 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.196934 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-podinfo\") pod \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " Oct 31 00:01:05.197213 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.197011 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-envoy\") pod \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " Oct 31 00:01:05.197213 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.197055 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-certs\") pod \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\" (UID: \"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8\") " Oct 31 00:01:05.197213 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.197051 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-data" pod "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" (UID: "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:05.197213 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.197065 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-socket" pod "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" (UID: "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:05.197213 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.197147 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-credential-socket" pod "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" (UID: "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:05.197417 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.197320 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-certs" pod "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" (UID: "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:05.197417 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.197365 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istiod-ca-cert" pod "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" (UID: "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:05.197722 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.197670 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:05.197722 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.197693 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:05.197722 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.197707 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:05.197722 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.197720 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:05.197884 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.197732 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:05.199412 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.199362 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-kube-api-access-5bkxr" pod "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" (UID: "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8"). InnerVolumeSpecName "kube-api-access-5bkxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:05.199541 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.199434 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-token" pod "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" (UID: "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:05.199756 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.199736 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-podinfo" pod "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" (UID: "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:01:05.199856 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.199774 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-envoy" pod "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" (UID: "1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:05.298577 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.298542 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:05.298577 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.298568 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bkxr\" (UniqueName: \"kubernetes.io/projected/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-kube-api-access-5bkxr\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:05.298577 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.298577 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:05.298577 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.298586 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:05.611983 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.611951 7570 generic.go:356] "Generic (PLEG): container finished" podID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" containerID="13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21" exitCode=0 Oct 31 00:01:05.612173 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.612007 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" Oct 31 00:01:05.612173 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.612038 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" event={"ID":"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8","Type":"ContainerDied","Data":"13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21"} Oct 31 00:01:05.612173 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.612103 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd" event={"ID":"1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8","Type":"ContainerDied","Data":"acc4e95013981419d8feb265350bbaa1f26b4997ff6516bb952280266c101fc7"} Oct 31 00:01:05.612173 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.612132 7570 scope.go:122] "RemoveContainer" containerID="13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21" Oct 31 00:01:05.624539 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.624522 7570 scope.go:122] "RemoveContainer" containerID="13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21" Oct 31 00:01:05.624851 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:05.624834 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21\": container with ID starting with 13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21 not found: ID does not exist" containerID="13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21" Oct 31 00:01:05.624905 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.624863 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21"} err="failed to get container status \"13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21\": rpc error: code = NotFound desc = could not find container \"13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21\": container with ID starting with 13090b098882ea491922053028fdd8f9e0926dce78da789ad451b5986258ea21 not found: ID does not exist" Oct 31 00:01:05.648420 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.648397 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd"] Oct 31 00:01:05.653393 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:05.653375 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-200408-openshift-default-7fdd497699-dvxbd"] Oct 31 00:01:06.270393 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:06.270361 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2"] Oct 31 00:01:06.603651 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:06.603618 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" path="/var/lib/kubelet/pods/1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8/volumes" Oct 31 00:01:09.671544 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.671496 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh"] Oct 31 00:01:09.672010 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.671754 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" containerID="cri-o://2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d" gracePeriod=30 Oct 31 00:01:09.925714 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.925651 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98"] Oct 31 00:01:09.928888 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.928864 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-885999-gtc98"] Oct 31 00:01:09.939158 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.939140 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 31 00:01:09.972778 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.972754 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-metrics-certs\") pod \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " Oct 31 00:01:09.972920 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.972795 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eedd4bdb-1495-47e6-884d-5ce26a05e72b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eedd4bdb-1495-47e6-884d-5ce26a05e72b-openshift-service-ca\") pod \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " Oct 31 00:01:09.972920 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.972887 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-maas-api-tls\") pod \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " Oct 31 00:01:09.973116 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.973096 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedd4bdb-1495-47e6-884d-5ce26a05e72b-kube-api-access-dljfh\" (UniqueName: \"kubernetes.io/projected/eedd4bdb-1495-47e6-884d-5ce26a05e72b-kube-api-access-dljfh\") pod \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\" (UID: \"eedd4bdb-1495-47e6-884d-5ce26a05e72b\") " Oct 31 00:01:09.973269 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.973242 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eedd4bdb-1495-47e6-884d-5ce26a05e72b-openshift-service-ca" pod "eedd4bdb-1495-47e6-884d-5ce26a05e72b" (UID: "eedd4bdb-1495-47e6-884d-5ce26a05e72b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:09.973639 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.973618 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eedd4bdb-1495-47e6-884d-5ce26a05e72b-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:09.975224 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.975197 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-maas-api-tls" pod "eedd4bdb-1495-47e6-884d-5ce26a05e72b" (UID: "eedd4bdb-1495-47e6-884d-5ce26a05e72b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:09.975788 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.975768 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedd4bdb-1495-47e6-884d-5ce26a05e72b-kube-api-access-dljfh" pod "eedd4bdb-1495-47e6-884d-5ce26a05e72b" (UID: "eedd4bdb-1495-47e6-884d-5ce26a05e72b"). InnerVolumeSpecName "kube-api-access-dljfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:09.976010 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:09.975990 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-metrics-certs" pod "eedd4bdb-1495-47e6-884d-5ce26a05e72b" (UID: "eedd4bdb-1495-47e6-884d-5ce26a05e72b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:10.075972 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.075944 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:10.075972 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.075968 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eedd4bdb-1495-47e6-884d-5ce26a05e72b-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:10.076106 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.076008 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dljfh\" (UniqueName: \"kubernetes.io/projected/eedd4bdb-1495-47e6-884d-5ce26a05e72b-kube-api-access-dljfh\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:10.462668 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.462633 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4"] Oct 31 00:01:10.602532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.602499 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4414480-554a-4a1b-b70a-49e75e243696" path="/var/lib/kubelet/pods/b4414480-554a-4a1b-b70a-49e75e243696/volumes" Oct 31 00:01:10.646149 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.646118 7570 generic.go:356] "Generic (PLEG): container finished" podID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerID="2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d" exitCode=0 Oct 31 00:01:10.646268 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.646156 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" Oct 31 00:01:10.646268 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.646199 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" event={"ID":"eedd4bdb-1495-47e6-884d-5ce26a05e72b","Type":"ContainerDied","Data":"2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d"} Oct 31 00:01:10.646268 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.646246 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh" event={"ID":"eedd4bdb-1495-47e6-884d-5ce26a05e72b","Type":"ContainerDied","Data":"c15f4cb49b41c79f4a364ec6d09bb2a1293706fbb896c110d39fd96de690d417"} Oct 31 00:01:10.646384 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.646272 7570 scope.go:122] "RemoveContainer" containerID="2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d" Oct 31 00:01:10.658026 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.658004 7570 scope.go:122] "RemoveContainer" containerID="e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358" Oct 31 00:01:10.668348 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.668332 7570 scope.go:122] "RemoveContainer" containerID="2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d" Oct 31 00:01:10.668638 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:10.668619 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d\": container with ID starting with 2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d not found: ID does not exist" containerID="2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d" Oct 31 00:01:10.668691 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.668645 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d"} err="failed to get container status \"2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d\": rpc error: code = NotFound desc = could not find container \"2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d\": container with ID starting with 2576b77bdbe613665be2dd65701ed7c9f883ff885ce39489541dd281e62cb38d not found: ID does not exist" Oct 31 00:01:10.668691 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.668661 7570 scope.go:122] "RemoveContainer" containerID="e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358" Oct 31 00:01:10.668909 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:10.668889 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358\": container with ID starting with e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358 not found: ID does not exist" containerID="e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358" Oct 31 00:01:10.668953 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.668916 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358"} err="failed to get container status \"e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358\": rpc error: code = NotFound desc = could not find container \"e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358\": container with ID starting with e9010f5ac6c6727bce0d88a79a22108311bbbcb51844fbabd408816b5ad29358 not found: ID does not exist" Oct 31 00:01:10.678450 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.678426 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh"] Oct 31 00:01:10.681576 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:10.681557 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-885999-75c9769ffc-mt4fh"] Oct 31 00:01:11.270832 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.270769 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" podUID="131867e6-7957-48d7-96d5-50f9e8ff45b2" containerName="payload-processing" containerID="cri-o://3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4" gracePeriod=25 Oct 31 00:01:11.540626 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.540567 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 31 00:01:11.593994 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.593962 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131867e6-7957-48d7-96d5-50f9e8ff45b2-kube-api-access-6nkjq\" (UniqueName: \"kubernetes.io/projected/131867e6-7957-48d7-96d5-50f9e8ff45b2-kube-api-access-6nkjq\") pod \"131867e6-7957-48d7-96d5-50f9e8ff45b2\" (UID: \"131867e6-7957-48d7-96d5-50f9e8ff45b2\") " Oct 31 00:01:11.594163 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.594024 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131867e6-7957-48d7-96d5-50f9e8ff45b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131867e6-7957-48d7-96d5-50f9e8ff45b2-plugins-config-volume\") pod \"131867e6-7957-48d7-96d5-50f9e8ff45b2\" (UID: \"131867e6-7957-48d7-96d5-50f9e8ff45b2\") " Oct 31 00:01:11.594542 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.594512 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131867e6-7957-48d7-96d5-50f9e8ff45b2-plugins-config-volume" pod "131867e6-7957-48d7-96d5-50f9e8ff45b2" (UID: "131867e6-7957-48d7-96d5-50f9e8ff45b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:11.596773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.596748 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131867e6-7957-48d7-96d5-50f9e8ff45b2-kube-api-access-6nkjq" pod "131867e6-7957-48d7-96d5-50f9e8ff45b2" (UID: "131867e6-7957-48d7-96d5-50f9e8ff45b2"). InnerVolumeSpecName "kube-api-access-6nkjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:11.655220 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.655192 7570 generic.go:356] "Generic (PLEG): container finished" podID="131867e6-7957-48d7-96d5-50f9e8ff45b2" containerID="3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4" exitCode=0 Oct 31 00:01:11.655220 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.655211 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" Oct 31 00:01:11.655393 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.655250 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" event={"ID":"131867e6-7957-48d7-96d5-50f9e8ff45b2","Type":"ContainerDied","Data":"3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4"} Oct 31 00:01:11.655393 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.655309 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2" event={"ID":"131867e6-7957-48d7-96d5-50f9e8ff45b2","Type":"ContainerDied","Data":"154cd04b7632922f6fa863643d7d35c2de9327718fbbeb3e91976dd01b49bae0"} Oct 31 00:01:11.655393 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.655331 7570 scope.go:122] "RemoveContainer" containerID="3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4" Oct 31 00:01:11.666841 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.666824 7570 scope.go:122] "RemoveContainer" containerID="3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4" Oct 31 00:01:11.667094 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:11.667073 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4\": container with ID starting with 3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4 not found: ID does not exist" containerID="3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4" Oct 31 00:01:11.667148 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.667102 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4"} err="failed to get container status \"3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4\": rpc error: code = NotFound desc = could not find container \"3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4\": container with ID starting with 3500da9112f0f8b68947468f1e6e0c6f1182adf5f84e649275d36569416ba1e4 not found: ID does not exist" Oct 31 00:01:11.684176 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.684145 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2"] Oct 31 00:01:11.688763 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.688741 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-885999-6d8d696d9f-d4gz2"] Oct 31 00:01:11.695873 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.695850 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nkjq\" (UniqueName: \"kubernetes.io/projected/131867e6-7957-48d7-96d5-50f9e8ff45b2-kube-api-access-6nkjq\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:11.695955 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:11.695874 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131867e6-7957-48d7-96d5-50f9e8ff45b2-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:12.601845 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:12.601814 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131867e6-7957-48d7-96d5-50f9e8ff45b2" path="/var/lib/kubelet/pods/131867e6-7957-48d7-96d5-50f9e8ff45b2/volumes" Oct 31 00:01:12.602255 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:12.602242 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" path="/var/lib/kubelet/pods/eedd4bdb-1495-47e6-884d-5ce26a05e72b/volumes" Oct 31 00:01:14.328858 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:14.328824 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t"] Oct 31 00:01:14.332749 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:14.332710 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-86d846-pb76t"] Oct 31 00:01:14.603080 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:14.603007 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f307a0d-2b1a-48ab-893c-7d983abab906" path="/var/lib/kubelet/pods/6f307a0d-2b1a-48ab-893c-7d983abab906/volumes" Oct 31 00:01:15.463507 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:15.463438 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" podUID="ca8613ba-3272-43d3-9661-33618f859cff" containerName="payload-pre-processing" containerID="cri-o://4f678db60e39015edd8c36718ebafbe5657d2770f28741621f2345eeaa93dc40" gracePeriod=25 Oct 31 00:01:15.688323 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:15.688294 7570 generic.go:356] "Generic (PLEG): container finished" podID="ca8613ba-3272-43d3-9661-33618f859cff" containerID="4f678db60e39015edd8c36718ebafbe5657d2770f28741621f2345eeaa93dc40" exitCode=0 Oct 31 00:01:15.688515 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:15.688365 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" event={"ID":"ca8613ba-3272-43d3-9661-33618f859cff","Type":"ContainerDied","Data":"4f678db60e39015edd8c36718ebafbe5657d2770f28741621f2345eeaa93dc40"} Oct 31 00:01:15.729958 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:15.729898 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 31 00:01:15.845375 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:15.845339 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8613ba-3272-43d3-9661-33618f859cff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8613ba-3272-43d3-9661-33618f859cff-plugins-config-volume\") pod \"ca8613ba-3272-43d3-9661-33618f859cff\" (UID: \"ca8613ba-3272-43d3-9661-33618f859cff\") " Oct 31 00:01:15.845577 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:15.845403 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8613ba-3272-43d3-9661-33618f859cff-kube-api-access-dq6pk\" (UniqueName: \"kubernetes.io/projected/ca8613ba-3272-43d3-9661-33618f859cff-kube-api-access-dq6pk\") pod \"ca8613ba-3272-43d3-9661-33618f859cff\" (UID: \"ca8613ba-3272-43d3-9661-33618f859cff\") " Oct 31 00:01:15.845760 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:15.845737 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8613ba-3272-43d3-9661-33618f859cff-plugins-config-volume" pod "ca8613ba-3272-43d3-9661-33618f859cff" (UID: "ca8613ba-3272-43d3-9661-33618f859cff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:15.845959 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:15.845943 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8613ba-3272-43d3-9661-33618f859cff-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:15.847757 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:15.847732 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8613ba-3272-43d3-9661-33618f859cff-kube-api-access-dq6pk" pod "ca8613ba-3272-43d3-9661-33618f859cff" (UID: "ca8613ba-3272-43d3-9661-33618f859cff"). InnerVolumeSpecName "kube-api-access-dq6pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:15.947484 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:15.947426 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq6pk\" (UniqueName: \"kubernetes.io/projected/ca8613ba-3272-43d3-9661-33618f859cff-kube-api-access-dq6pk\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:16.696515 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:16.696484 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" Oct 31 00:01:16.696515 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:16.696496 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4" event={"ID":"ca8613ba-3272-43d3-9661-33618f859cff","Type":"ContainerDied","Data":"659eac5f22524fc511fe71b153e5eb46fe25bf6c1e3d6abc41dcecd917bcdfd0"} Oct 31 00:01:16.697030 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:16.696552 7570 scope.go:122] "RemoveContainer" containerID="4f678db60e39015edd8c36718ebafbe5657d2770f28741621f2345eeaa93dc40" Oct 31 00:01:16.771043 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:16.771016 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4"] Oct 31 00:01:16.794150 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:16.794122 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-86d846-666d656b47-h7sb4"] Oct 31 00:01:18.595098 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595066 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8"] Oct 31 00:01:18.595619 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595604 7570 cpu_manager.go:414] "containerMap: removing container" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" Oct 31 00:01:18.595672 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595620 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" Oct 31 00:01:18.595672 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595635 7570 cpu_manager.go:414] "containerMap: removing container" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" Oct 31 00:01:18.595672 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595640 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" Oct 31 00:01:18.595672 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595655 7570 cpu_manager.go:414] "containerMap: removing container" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" Oct 31 00:01:18.595672 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595660 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" Oct 31 00:01:18.595672 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595669 7570 cpu_manager.go:414] "containerMap: removing container" podUID="131867e6-7957-48d7-96d5-50f9e8ff45b2" containerName="payload-processing" Oct 31 00:01:18.595837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595675 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131867e6-7957-48d7-96d5-50f9e8ff45b2" containerName="payload-processing" Oct 31 00:01:18.595837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595681 7570 cpu_manager.go:414] "containerMap: removing container" podUID="6f307a0d-2b1a-48ab-893c-7d983abab906" containerName="revoke-keys" Oct 31 00:01:18.595837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595686 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f307a0d-2b1a-48ab-893c-7d983abab906" containerName="revoke-keys" Oct 31 00:01:18.595837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595701 7570 cpu_manager.go:414] "containerMap: removing container" podUID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" containerName="istio-proxy" Oct 31 00:01:18.595837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595705 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" containerName="istio-proxy" Oct 31 00:01:18.595837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595724 7570 cpu_manager.go:414] "containerMap: removing container" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" Oct 31 00:01:18.595837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595729 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" Oct 31 00:01:18.595837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595741 7570 cpu_manager.go:414] "containerMap: removing container" podUID="ca8613ba-3272-43d3-9661-33618f859cff" containerName="payload-pre-processing" Oct 31 00:01:18.595837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595746 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8613ba-3272-43d3-9661-33618f859cff" containerName="payload-pre-processing" Oct 31 00:01:18.595837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595826 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8613ba-3272-43d3-9661-33618f859cff" containerName="payload-pre-processing" Oct 31 00:01:18.595837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595836 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" Oct 31 00:01:18.596122 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595845 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="eedd4bdb-1495-47e6-884d-5ce26a05e72b" containerName="maas-api" Oct 31 00:01:18.596122 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595853 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c25f85-fbde-4c00-94c9-502637e57ba5" containerName="istio-proxy" Oct 31 00:01:18.596122 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595858 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="131867e6-7957-48d7-96d5-50f9e8ff45b2" containerName="payload-processing" Oct 31 00:01:18.596122 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595863 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f307a0d-2b1a-48ab-893c-7d983abab906" containerName="revoke-keys" Oct 31 00:01:18.596122 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595871 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b57f5ed-88f7-4eb3-a209-3b0e24dd33f8" containerName="istio-proxy" Oct 31 00:01:18.596122 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.595879 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74b8dd0-7067-44fd-94c0-948568ca285c" containerName="istio-proxy" Oct 31 00:01:18.600290 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.600267 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:18.609476 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.609434 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8613ba-3272-43d3-9661-33618f859cff" path="/var/lib/kubelet/pods/ca8613ba-3272-43d3-9661-33618f859cff/volumes" Oct 31 00:01:18.609835 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.609817 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8"] Oct 31 00:01:18.637516 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.637489 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:01:18.637701 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.637681 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:01:18.680258 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.680226 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8\" (UID: \"4ce6838f-955e-4414-8815-96fdf8880aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:18.680398 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.680265 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8\" (UID: \"4ce6838f-955e-4414-8815-96fdf8880aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:18.680444 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.680419 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k4ml\" (UniqueName: \"kubernetes.io/projected/4ce6838f-955e-4414-8815-96fdf8880aec-kube-api-access-5k4ml\") pod \"maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8\" (UID: \"4ce6838f-955e-4414-8815-96fdf8880aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:18.781638 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.781600 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k4ml\" (UniqueName: \"kubernetes.io/projected/4ce6838f-955e-4414-8815-96fdf8880aec-kube-api-access-5k4ml\") pod \"maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8\" (UID: \"4ce6838f-955e-4414-8815-96fdf8880aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:18.781793 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.781759 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8\" (UID: \"4ce6838f-955e-4414-8815-96fdf8880aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:18.781793 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.781788 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8\" (UID: \"4ce6838f-955e-4414-8815-96fdf8880aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:18.782450 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.782427 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8\" (UID: \"4ce6838f-955e-4414-8815-96fdf8880aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:18.784219 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.784195 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8\" (UID: \"4ce6838f-955e-4414-8815-96fdf8880aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:18.796359 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.796341 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k4ml\" (UniqueName: \"kubernetes.io/projected/4ce6838f-955e-4414-8815-96fdf8880aec-kube-api-access-5k4ml\") pod \"maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8\" (UID: \"4ce6838f-955e-4414-8815-96fdf8880aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:18.919273 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:18.919196 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:19.235548 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:19.234697 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8"] Oct 31 00:01:19.722903 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:19.722866 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" event={"ID":"4ce6838f-955e-4414-8815-96fdf8880aec","Type":"ContainerStarted","Data":"43264f2db4aedb9084176733efe7533377dff1dcb6a22d7d4869e437c4693da3"} Oct 31 00:01:19.723327 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:19.722927 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" event={"ID":"4ce6838f-955e-4414-8815-96fdf8880aec","Type":"ContainerStarted","Data":"48810fa30bff4715ddcd7abb83c74510165c2c30d19373b9cdb8b61ba83041c6"} Oct 31 00:01:19.743790 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:19.743742 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" podStartSLOduration=1.743725814 podStartE2EDuration="1.743725814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:19.740923954 +0000 UTC m=+2491.259925060" watchObservedRunningTime="2026-10-31 00:01:19.743725814 +0000 UTC m=+2491.262726904" Oct 31 00:01:20.731223 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:20.731193 7570 generic.go:356] "Generic (PLEG): container finished" podID="4ce6838f-955e-4414-8815-96fdf8880aec" containerID="43264f2db4aedb9084176733efe7533377dff1dcb6a22d7d4869e437c4693da3" exitCode=0 Oct 31 00:01:20.731630 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:20.731288 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" event={"ID":"4ce6838f-955e-4414-8815-96fdf8880aec","Type":"ContainerDied","Data":"43264f2db4aedb9084176733efe7533377dff1dcb6a22d7d4869e437c4693da3"} Oct 31 00:01:21.898069 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:21.898046 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:22.032709 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.032631 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-openshift-ca\") pod \"4ce6838f-955e-4414-8815-96fdf8880aec\" (UID: \"4ce6838f-955e-4414-8815-96fdf8880aec\") " Oct 31 00:01:22.032709 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.032688 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-xks-ca\") pod \"4ce6838f-955e-4414-8815-96fdf8880aec\" (UID: \"4ce6838f-955e-4414-8815-96fdf8880aec\") " Oct 31 00:01:22.032876 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.032823 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ce6838f-955e-4414-8815-96fdf8880aec-kube-api-access-5k4ml\" (UniqueName: \"kubernetes.io/projected/4ce6838f-955e-4414-8815-96fdf8880aec-kube-api-access-5k4ml\") pod \"4ce6838f-955e-4414-8815-96fdf8880aec\" (UID: \"4ce6838f-955e-4414-8815-96fdf8880aec\") " Oct 31 00:01:22.033081 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.033055 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-openshift-ca" pod "4ce6838f-955e-4414-8815-96fdf8880aec" (UID: "4ce6838f-955e-4414-8815-96fdf8880aec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:22.033415 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.033330 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:22.035017 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.034989 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-xks-ca" pod "4ce6838f-955e-4414-8815-96fdf8880aec" (UID: "4ce6838f-955e-4414-8815-96fdf8880aec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:22.035143 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.035125 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce6838f-955e-4414-8815-96fdf8880aec-kube-api-access-5k4ml" pod "4ce6838f-955e-4414-8815-96fdf8880aec" (UID: "4ce6838f-955e-4414-8815-96fdf8880aec"). InnerVolumeSpecName "kube-api-access-5k4ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:22.134170 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.134139 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce6838f-955e-4414-8815-96fdf8880aec-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:22.134170 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.134165 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k4ml\" (UniqueName: \"kubernetes.io/projected/4ce6838f-955e-4414-8815-96fdf8880aec-kube-api-access-5k4ml\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:22.544699 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.544663 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb"] Oct 31 00:01:22.545316 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.545288 7570 cpu_manager.go:414] "containerMap: removing container" podUID="4ce6838f-955e-4414-8815-96fdf8880aec" containerName="revoke-keys" Oct 31 00:01:22.545316 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.545316 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ce6838f-955e-4414-8815-96fdf8880aec" containerName="revoke-keys" Oct 31 00:01:22.545534 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.545496 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ce6838f-955e-4414-8815-96fdf8880aec" containerName="revoke-keys" Oct 31 00:01:22.549137 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.549119 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:22.556785 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.556756 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb"] Oct 31 00:01:22.643221 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.643191 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb\" (UID: \"95338565-81e8-4345-ba7a-d74ad08694a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:22.643380 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.643251 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltpzn\" (UniqueName: \"kubernetes.io/projected/95338565-81e8-4345-ba7a-d74ad08694a8-kube-api-access-ltpzn\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb\" (UID: \"95338565-81e8-4345-ba7a-d74ad08694a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:22.643422 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.643381 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb\" (UID: \"95338565-81e8-4345-ba7a-d74ad08694a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:22.744119 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.744090 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb\" (UID: \"95338565-81e8-4345-ba7a-d74ad08694a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:22.744308 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.744149 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltpzn\" (UniqueName: \"kubernetes.io/projected/95338565-81e8-4345-ba7a-d74ad08694a8-kube-api-access-ltpzn\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb\" (UID: \"95338565-81e8-4345-ba7a-d74ad08694a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:22.744308 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.744234 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb\" (UID: \"95338565-81e8-4345-ba7a-d74ad08694a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:22.745039 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.745012 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb\" (UID: \"95338565-81e8-4345-ba7a-d74ad08694a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:22.746042 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.746012 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" event={"ID":"4ce6838f-955e-4414-8815-96fdf8880aec","Type":"ContainerDied","Data":"48810fa30bff4715ddcd7abb83c74510165c2c30d19373b9cdb8b61ba83041c6"} Oct 31 00:01:22.746140 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.746041 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8" Oct 31 00:01:22.746140 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.746072 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48810fa30bff4715ddcd7abb83c74510165c2c30d19373b9cdb8b61ba83041c6" Oct 31 00:01:22.746928 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.746908 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb\" (UID: \"95338565-81e8-4345-ba7a-d74ad08694a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:22.756691 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.756668 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltpzn\" (UniqueName: \"kubernetes.io/projected/95338565-81e8-4345-ba7a-d74ad08694a8-kube-api-access-ltpzn\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb\" (UID: \"95338565-81e8-4345-ba7a-d74ad08694a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:22.874617 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:22.874582 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:23.157929 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:23.157897 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb"] Oct 31 00:01:23.159139 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:01:23.159103 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95338565_81e8_4345_ba7a_d74ad08694a8.slice/crio-e0c72fd7ef531e82cf6b1089f809aba79cfff8df9cd692ecb54a7c95e410b4de WatchSource:0}: Error finding container e0c72fd7ef531e82cf6b1089f809aba79cfff8df9cd692ecb54a7c95e410b4de: Status 404 returned error can't find the container with id e0c72fd7ef531e82cf6b1089f809aba79cfff8df9cd692ecb54a7c95e410b4de Oct 31 00:01:23.402928 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:23.402898 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb"] Oct 31 00:01:23.417916 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:23.417853 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl"] Oct 31 00:01:23.430556 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:23.430520 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4"] Oct 31 00:01:23.755727 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:23.755638 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" event={"ID":"95338565-81e8-4345-ba7a-d74ad08694a8","Type":"ContainerStarted","Data":"9964ef3208428295b2be8c0118a49764128634711070f478dadbb40176e57fe6"} Oct 31 00:01:23.755727 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:23.755692 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" event={"ID":"95338565-81e8-4345-ba7a-d74ad08694a8","Type":"ContainerStarted","Data":"e0c72fd7ef531e82cf6b1089f809aba79cfff8df9cd692ecb54a7c95e410b4de"} Oct 31 00:01:23.810060 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:23.810011 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" podStartSLOduration=1.809997771 podStartE2EDuration="1.809997771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:23.807724317 +0000 UTC m=+2495.326725408" watchObservedRunningTime="2026-10-31 00:01:23.809997771 +0000 UTC m=+2495.328998860" Oct 31 00:01:24.763487 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:24.763426 7570 generic.go:356] "Generic (PLEG): container finished" podID="95338565-81e8-4345-ba7a-d74ad08694a8" containerID="9964ef3208428295b2be8c0118a49764128634711070f478dadbb40176e57fe6" exitCode=0 Oct 31 00:01:24.763868 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:24.763516 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" event={"ID":"95338565-81e8-4345-ba7a-d74ad08694a8","Type":"ContainerDied","Data":"9964ef3208428295b2be8c0118a49764128634711070f478dadbb40176e57fe6"} Oct 31 00:01:25.938378 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:25.938354 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:26.091864 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:26.091832 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-xks-ca\") pod \"95338565-81e8-4345-ba7a-d74ad08694a8\" (UID: \"95338565-81e8-4345-ba7a-d74ad08694a8\") " Oct 31 00:01:26.092015 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:26.091930 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-openshift-ca\") pod \"95338565-81e8-4345-ba7a-d74ad08694a8\" (UID: \"95338565-81e8-4345-ba7a-d74ad08694a8\") " Oct 31 00:01:26.092102 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:26.092090 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95338565-81e8-4345-ba7a-d74ad08694a8-kube-api-access-ltpzn\" (UniqueName: \"kubernetes.io/projected/95338565-81e8-4345-ba7a-d74ad08694a8-kube-api-access-ltpzn\") pod \"95338565-81e8-4345-ba7a-d74ad08694a8\" (UID: \"95338565-81e8-4345-ba7a-d74ad08694a8\") " Oct 31 00:01:26.092323 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:26.092294 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-openshift-ca" pod "95338565-81e8-4345-ba7a-d74ad08694a8" (UID: "95338565-81e8-4345-ba7a-d74ad08694a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:26.092502 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:26.092481 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:26.094212 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:26.094187 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-xks-ca" pod "95338565-81e8-4345-ba7a-d74ad08694a8" (UID: "95338565-81e8-4345-ba7a-d74ad08694a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:26.094326 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:26.094306 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95338565-81e8-4345-ba7a-d74ad08694a8-kube-api-access-ltpzn" pod "95338565-81e8-4345-ba7a-d74ad08694a8" (UID: "95338565-81e8-4345-ba7a-d74ad08694a8"). InnerVolumeSpecName "kube-api-access-ltpzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:26.193334 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:26.193304 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltpzn\" (UniqueName: \"kubernetes.io/projected/95338565-81e8-4345-ba7a-d74ad08694a8-kube-api-access-ltpzn\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:26.193334 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:26.193332 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95338565-81e8-4345-ba7a-d74ad08694a8-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:26.782323 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:26.782294 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" Oct 31 00:01:26.782512 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:26.782332 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb" event={"ID":"95338565-81e8-4345-ba7a-d74ad08694a8","Type":"ContainerDied","Data":"e0c72fd7ef531e82cf6b1089f809aba79cfff8df9cd692ecb54a7c95e410b4de"} Oct 31 00:01:26.782512 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:26.782379 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0c72fd7ef531e82cf6b1089f809aba79cfff8df9cd692ecb54a7c95e410b4de" Oct 31 00:01:27.376340 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:27.376304 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs"] Oct 31 00:01:28.716341 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:28.716306 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb"] Oct 31 00:01:28.726685 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:28.726662 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm"] Oct 31 00:01:29.411953 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:29.411912 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:01:33.129737 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.129704 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh"] Oct 31 00:01:33.130420 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.130403 7570 cpu_manager.go:414] "containerMap: removing container" podUID="95338565-81e8-4345-ba7a-d74ad08694a8" containerName="revoke-keys" Oct 31 00:01:33.130500 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.130423 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95338565-81e8-4345-ba7a-d74ad08694a8" containerName="revoke-keys" Oct 31 00:01:33.130556 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.130539 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="95338565-81e8-4345-ba7a-d74ad08694a8" containerName="revoke-keys" Oct 31 00:01:33.134274 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.134256 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:33.143550 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.143513 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:01:33.143684 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.143601 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:01:33.143684 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.143630 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4"] Oct 31 00:01:33.148117 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.148096 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:33.148604 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.148581 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh"] Oct 31 00:01:33.162105 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.162079 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4"] Oct 31 00:01:33.179491 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.179442 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxq7\" (UniqueName: \"kubernetes.io/projected/1ab1e878-bca9-41c4-81e6-0001487f5ff1-kube-api-access-tdxq7\") pod \"maas-api-revoke-sub-e64d94e2-wl5kh\" (UID: \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:33.179622 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.179515 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7af98d5a-m66c4\" (UID: \"28216477-7eed-4e96-a7e7-262ab1d60bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:33.179691 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.179676 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7af98d5a-m66c4\" (UID: \"28216477-7eed-4e96-a7e7-262ab1d60bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:33.179755 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.179739 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e64d94e2-wl5kh\" (UID: \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:33.179804 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.179792 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e64d94e2-wl5kh\" (UID: \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:33.179845 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.179832 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvrn5\" (UniqueName: \"kubernetes.io/projected/28216477-7eed-4e96-a7e7-262ab1d60bb9-kube-api-access-dvrn5\") pod \"maas-api-revoke-sub-7af98d5a-m66c4\" (UID: \"28216477-7eed-4e96-a7e7-262ab1d60bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:33.197134 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.197102 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6"] Oct 31 00:01:33.202438 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.202413 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:33.208532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.208432 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6"] Oct 31 00:01:33.280573 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.280543 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvrn5\" (UniqueName: \"kubernetes.io/projected/28216477-7eed-4e96-a7e7-262ab1d60bb9-kube-api-access-dvrn5\") pod \"maas-api-revoke-sub-7af98d5a-m66c4\" (UID: \"28216477-7eed-4e96-a7e7-262ab1d60bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:33.280735 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.280613 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxq7\" (UniqueName: \"kubernetes.io/projected/1ab1e878-bca9-41c4-81e6-0001487f5ff1-kube-api-access-tdxq7\") pod \"maas-api-revoke-sub-e64d94e2-wl5kh\" (UID: \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:33.280735 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.280644 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjzlh\" (UniqueName: \"kubernetes.io/projected/47465535-f16b-4864-a764-71f07c0d273d-kube-api-access-gjzlh\") pod \"maas-api-revoke-sub-9fbcbc45-htxw6\" (UID: \"47465535-f16b-4864-a764-71f07c0d273d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:33.280735 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.280675 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7af98d5a-m66c4\" (UID: \"28216477-7eed-4e96-a7e7-262ab1d60bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:33.280888 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.280784 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47465535-f16b-4864-a764-71f07c0d273d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fbcbc45-htxw6\" (UID: \"47465535-f16b-4864-a764-71f07c0d273d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:33.280888 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.280831 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47465535-f16b-4864-a764-71f07c0d273d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fbcbc45-htxw6\" (UID: \"47465535-f16b-4864-a764-71f07c0d273d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:33.280888 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.280862 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7af98d5a-m66c4\" (UID: \"28216477-7eed-4e96-a7e7-262ab1d60bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:33.281039 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.280949 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e64d94e2-wl5kh\" (UID: \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:33.281039 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.280981 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e64d94e2-wl5kh\" (UID: \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:33.281453 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.281433 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7af98d5a-m66c4\" (UID: \"28216477-7eed-4e96-a7e7-262ab1d60bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:33.281638 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.281620 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e64d94e2-wl5kh\" (UID: \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:33.283333 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.283315 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7af98d5a-m66c4\" (UID: \"28216477-7eed-4e96-a7e7-262ab1d60bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:33.283418 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.283355 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e64d94e2-wl5kh\" (UID: \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:33.292781 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.292760 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxq7\" (UniqueName: \"kubernetes.io/projected/1ab1e878-bca9-41c4-81e6-0001487f5ff1-kube-api-access-tdxq7\") pod \"maas-api-revoke-sub-e64d94e2-wl5kh\" (UID: \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:33.292947 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.292930 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvrn5\" (UniqueName: \"kubernetes.io/projected/28216477-7eed-4e96-a7e7-262ab1d60bb9-kube-api-access-dvrn5\") pod \"maas-api-revoke-sub-7af98d5a-m66c4\" (UID: \"28216477-7eed-4e96-a7e7-262ab1d60bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:33.382549 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.382450 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjzlh\" (UniqueName: \"kubernetes.io/projected/47465535-f16b-4864-a764-71f07c0d273d-kube-api-access-gjzlh\") pod \"maas-api-revoke-sub-9fbcbc45-htxw6\" (UID: \"47465535-f16b-4864-a764-71f07c0d273d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:33.382673 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.382567 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47465535-f16b-4864-a764-71f07c0d273d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fbcbc45-htxw6\" (UID: \"47465535-f16b-4864-a764-71f07c0d273d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:33.382673 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.382621 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47465535-f16b-4864-a764-71f07c0d273d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fbcbc45-htxw6\" (UID: \"47465535-f16b-4864-a764-71f07c0d273d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:33.383318 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.383288 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47465535-f16b-4864-a764-71f07c0d273d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fbcbc45-htxw6\" (UID: \"47465535-f16b-4864-a764-71f07c0d273d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:33.385683 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.385642 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47465535-f16b-4864-a764-71f07c0d273d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fbcbc45-htxw6\" (UID: \"47465535-f16b-4864-a764-71f07c0d273d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:33.399365 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.399337 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjzlh\" (UniqueName: \"kubernetes.io/projected/47465535-f16b-4864-a764-71f07c0d273d-kube-api-access-gjzlh\") pod \"maas-api-revoke-sub-9fbcbc45-htxw6\" (UID: \"47465535-f16b-4864-a764-71f07c0d273d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:33.450903 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.450872 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:33.470182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.470152 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:33.525125 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.522337 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:33.717038 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.716977 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerName="payload-processing" containerID="cri-o://c43dd4085f8264bf875fd779a96aad3200de0e9c03ed69d1979990a51aa50960" gracePeriod=25 Oct 31 00:01:33.727552 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.727512 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" podUID="1e8a6ea5-1c91-462f-94fd-7136a48971e9" containerName="payload-pre-processing" containerID="cri-o://cc6af65b1d32bcfd65a99cdbd83d31bf7073c45b98d943e624c0632032ffcd9f" gracePeriod=25 Oct 31 00:01:33.744137 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.744108 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh"] Oct 31 00:01:33.745305 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:01:33.745283 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab1e878_bca9_41c4_81e6_0001487f5ff1.slice/crio-da01cb37dd692c7619696351f23237a1bb0bb7fcbb59c980883f00bbb7216115 WatchSource:0}: Error finding container da01cb37dd692c7619696351f23237a1bb0bb7fcbb59c980883f00bbb7216115: Status 404 returned error can't find the container with id da01cb37dd692c7619696351f23237a1bb0bb7fcbb59c980883f00bbb7216115 Oct 31 00:01:33.769327 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.769305 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4"] Oct 31 00:01:33.779804 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:01:33.779778 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28216477_7eed_4e96_a7e7_262ab1d60bb9.slice/crio-4696a0b95c94257866b0421199966656961b8d59da9ddad9454484eb27036f04 WatchSource:0}: Error finding container 4696a0b95c94257866b0421199966656961b8d59da9ddad9454484eb27036f04: Status 404 returned error can't find the container with id 4696a0b95c94257866b0421199966656961b8d59da9ddad9454484eb27036f04 Oct 31 00:01:33.840599 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.840566 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" event={"ID":"1ab1e878-bca9-41c4-81e6-0001487f5ff1","Type":"ContainerStarted","Data":"da01cb37dd692c7619696351f23237a1bb0bb7fcbb59c980883f00bbb7216115"} Oct 31 00:01:33.841893 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.841863 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" event={"ID":"28216477-7eed-4e96-a7e7-262ab1d60bb9","Type":"ContainerStarted","Data":"4696a0b95c94257866b0421199966656961b8d59da9ddad9454484eb27036f04"} Oct 31 00:01:33.848964 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:33.848939 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6"] Oct 31 00:01:33.849892 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:01:33.849868 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47465535_f16b_4864_a764_71f07c0d273d.slice/crio-8a032da19d272a97ef6e62abb8fb54f780b211486d535ee1fcb49545cf085ff4 WatchSource:0}: Error finding container 8a032da19d272a97ef6e62abb8fb54f780b211486d535ee1fcb49545cf085ff4: Status 404 returned error can't find the container with id 8a032da19d272a97ef6e62abb8fb54f780b211486d535ee1fcb49545cf085ff4 Oct 31 00:01:34.851368 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:34.851267 7570 generic.go:356] "Generic (PLEG): container finished" podID="1ab1e878-bca9-41c4-81e6-0001487f5ff1" containerID="58f6be62a832dd344eb19ea039c4c9fe57e3663e1acd371b9a3bc84465006187" exitCode=0 Oct 31 00:01:34.851864 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:34.851358 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" event={"ID":"1ab1e878-bca9-41c4-81e6-0001487f5ff1","Type":"ContainerDied","Data":"58f6be62a832dd344eb19ea039c4c9fe57e3663e1acd371b9a3bc84465006187"} Oct 31 00:01:34.853137 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:34.853112 7570 generic.go:356] "Generic (PLEG): container finished" podID="28216477-7eed-4e96-a7e7-262ab1d60bb9" containerID="5876d9f97e9a3a4ba40ef8652bca05ec8e8328294e24a88c46625df1f944cb70" exitCode=0 Oct 31 00:01:34.853246 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:34.853205 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" event={"ID":"28216477-7eed-4e96-a7e7-262ab1d60bb9","Type":"ContainerDied","Data":"5876d9f97e9a3a4ba40ef8652bca05ec8e8328294e24a88c46625df1f944cb70"} Oct 31 00:01:34.854907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:34.854885 7570 generic.go:356] "Generic (PLEG): container finished" podID="47465535-f16b-4864-a764-71f07c0d273d" containerID="66f8a656b3c8f7d5b9ae5d9e1d1fc8b464e1259845fa619d4bb353f63cd80d96" exitCode=0 Oct 31 00:01:34.854991 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:34.854968 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" event={"ID":"47465535-f16b-4864-a764-71f07c0d273d","Type":"ContainerDied","Data":"66f8a656b3c8f7d5b9ae5d9e1d1fc8b464e1259845fa619d4bb353f63cd80d96"} Oct 31 00:01:34.855033 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:34.855007 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" event={"ID":"47465535-f16b-4864-a764-71f07c0d273d","Type":"ContainerStarted","Data":"8a032da19d272a97ef6e62abb8fb54f780b211486d535ee1fcb49545cf085ff4"} Oct 31 00:01:36.075509 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.075437 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:36.114676 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.114603 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-openshift-ca\") pod \"28216477-7eed-4e96-a7e7-262ab1d60bb9\" (UID: \"28216477-7eed-4e96-a7e7-262ab1d60bb9\") " Oct 31 00:01:36.114804 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.114752 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-xks-ca\") pod \"28216477-7eed-4e96-a7e7-262ab1d60bb9\" (UID: \"28216477-7eed-4e96-a7e7-262ab1d60bb9\") " Oct 31 00:01:36.114961 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.114941 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28216477-7eed-4e96-a7e7-262ab1d60bb9-kube-api-access-dvrn5\" (UniqueName: \"kubernetes.io/projected/28216477-7eed-4e96-a7e7-262ab1d60bb9-kube-api-access-dvrn5\") pod \"28216477-7eed-4e96-a7e7-262ab1d60bb9\" (UID: \"28216477-7eed-4e96-a7e7-262ab1d60bb9\") " Oct 31 00:01:36.116353 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.116296 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-openshift-ca" pod "28216477-7eed-4e96-a7e7-262ab1d60bb9" (UID: "28216477-7eed-4e96-a7e7-262ab1d60bb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:36.118440 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.118415 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28216477-7eed-4e96-a7e7-262ab1d60bb9-kube-api-access-dvrn5" pod "28216477-7eed-4e96-a7e7-262ab1d60bb9" (UID: "28216477-7eed-4e96-a7e7-262ab1d60bb9"). InnerVolumeSpecName "kube-api-access-dvrn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:36.118851 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.118819 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-xks-ca" pod "28216477-7eed-4e96-a7e7-262ab1d60bb9" (UID: "28216477-7eed-4e96-a7e7-262ab1d60bb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:36.119081 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.119061 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:36.140789 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.140760 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:36.216249 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.216216 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-openshift-ca\") pod \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\" (UID: \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\") " Oct 31 00:01:36.216430 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.216315 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-xks-ca\") pod \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\" (UID: \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\") " Oct 31 00:01:36.216430 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.216367 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47465535-f16b-4864-a764-71f07c0d273d-kube-api-access-gjzlh\" (UniqueName: \"kubernetes.io/projected/47465535-f16b-4864-a764-71f07c0d273d-kube-api-access-gjzlh\") pod \"47465535-f16b-4864-a764-71f07c0d273d\" (UID: \"47465535-f16b-4864-a764-71f07c0d273d\") " Oct 31 00:01:36.216430 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.216413 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab1e878-bca9-41c4-81e6-0001487f5ff1-kube-api-access-tdxq7\" (UniqueName: \"kubernetes.io/projected/1ab1e878-bca9-41c4-81e6-0001487f5ff1-kube-api-access-tdxq7\") pod \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\" (UID: \"1ab1e878-bca9-41c4-81e6-0001487f5ff1\") " Oct 31 00:01:36.216622 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.216525 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47465535-f16b-4864-a764-71f07c0d273d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47465535-f16b-4864-a764-71f07c0d273d-maas-api-xks-ca\") pod \"47465535-f16b-4864-a764-71f07c0d273d\" (UID: \"47465535-f16b-4864-a764-71f07c0d273d\") " Oct 31 00:01:36.216715 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.216682 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-openshift-ca" pod "1ab1e878-bca9-41c4-81e6-0001487f5ff1" (UID: "1ab1e878-bca9-41c4-81e6-0001487f5ff1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:36.217575 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.217019 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47465535-f16b-4864-a764-71f07c0d273d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47465535-f16b-4864-a764-71f07c0d273d-maas-api-openshift-ca\") pod \"47465535-f16b-4864-a764-71f07c0d273d\" (UID: \"47465535-f16b-4864-a764-71f07c0d273d\") " Oct 31 00:01:36.218053 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.218028 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47465535-f16b-4864-a764-71f07c0d273d-maas-api-openshift-ca" pod "47465535-f16b-4864-a764-71f07c0d273d" (UID: "47465535-f16b-4864-a764-71f07c0d273d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:36.218697 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.218665 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47465535-f16b-4864-a764-71f07c0d273d-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:36.218697 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.218693 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvrn5\" (UniqueName: \"kubernetes.io/projected/28216477-7eed-4e96-a7e7-262ab1d60bb9-kube-api-access-dvrn5\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:36.218859 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.218723 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:36.218859 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.218742 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:36.218859 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.218758 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28216477-7eed-4e96-a7e7-262ab1d60bb9-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:36.219122 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.219098 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-xks-ca" pod "1ab1e878-bca9-41c4-81e6-0001487f5ff1" (UID: "1ab1e878-bca9-41c4-81e6-0001487f5ff1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:36.219386 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.219343 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab1e878-bca9-41c4-81e6-0001487f5ff1-kube-api-access-tdxq7" pod "1ab1e878-bca9-41c4-81e6-0001487f5ff1" (UID: "1ab1e878-bca9-41c4-81e6-0001487f5ff1"). InnerVolumeSpecName "kube-api-access-tdxq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:36.219772 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.219743 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47465535-f16b-4864-a764-71f07c0d273d-kube-api-access-gjzlh" pod "47465535-f16b-4864-a764-71f07c0d273d" (UID: "47465535-f16b-4864-a764-71f07c0d273d"). InnerVolumeSpecName "kube-api-access-gjzlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:36.220660 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.220638 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47465535-f16b-4864-a764-71f07c0d273d-maas-api-xks-ca" pod "47465535-f16b-4864-a764-71f07c0d273d" (UID: "47465535-f16b-4864-a764-71f07c0d273d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:36.320387 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.320353 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab1e878-bca9-41c4-81e6-0001487f5ff1-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:36.320387 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.320386 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjzlh\" (UniqueName: \"kubernetes.io/projected/47465535-f16b-4864-a764-71f07c0d273d-kube-api-access-gjzlh\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:36.320604 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.320399 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdxq7\" (UniqueName: \"kubernetes.io/projected/1ab1e878-bca9-41c4-81e6-0001487f5ff1-kube-api-access-tdxq7\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:36.320604 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.320408 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47465535-f16b-4864-a764-71f07c0d273d-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:36.869800 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.869709 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" Oct 31 00:01:36.869800 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.869702 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4" event={"ID":"28216477-7eed-4e96-a7e7-262ab1d60bb9","Type":"ContainerDied","Data":"4696a0b95c94257866b0421199966656961b8d59da9ddad9454484eb27036f04"} Oct 31 00:01:36.869999 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.869873 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4696a0b95c94257866b0421199966656961b8d59da9ddad9454484eb27036f04" Oct 31 00:01:36.871184 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.871167 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" Oct 31 00:01:36.871323 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.871197 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6" event={"ID":"47465535-f16b-4864-a764-71f07c0d273d","Type":"ContainerDied","Data":"8a032da19d272a97ef6e62abb8fb54f780b211486d535ee1fcb49545cf085ff4"} Oct 31 00:01:36.871323 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.871245 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a032da19d272a97ef6e62abb8fb54f780b211486d535ee1fcb49545cf085ff4" Oct 31 00:01:36.872950 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.872930 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" event={"ID":"1ab1e878-bca9-41c4-81e6-0001487f5ff1","Type":"ContainerDied","Data":"da01cb37dd692c7619696351f23237a1bb0bb7fcbb59c980883f00bbb7216115"} Oct 31 00:01:36.873045 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.872966 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh" Oct 31 00:01:36.873045 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:36.872970 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da01cb37dd692c7619696351f23237a1bb0bb7fcbb59c980883f00bbb7216115" Oct 31 00:01:38.425379 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.425328 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" containerID="cri-o://55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c" gracePeriod=45 Oct 31 00:01:38.439831 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.439790 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" containerID="cri-o://f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2" gracePeriod=45 Oct 31 00:01:38.451741 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.451700 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" containerID="cri-o://46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5" gracePeriod=45 Oct 31 00:01:38.468103 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.468076 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": dial tcp 10.134.0.72:8000: connect: connection refused" Oct 31 00:01:38.775688 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.775562 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 31 00:01:38.786404 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.786382 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 31 00:01:38.788566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.788548 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 31 00:01:38.853888 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.853857 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-dshm\") pod \"3e57b4d8-7b73-4490-9138-de9401d6fd65\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " Oct 31 00:01:38.853888 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.853904 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08cd1e28-c399-4cba-951e-0210a978109f-kube-api-access-cklbs\" (UniqueName: \"kubernetes.io/projected/08cd1e28-c399-4cba-951e-0210a978109f-kube-api-access-cklbs\") pod \"08cd1e28-c399-4cba-951e-0210a978109f\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " Oct 31 00:01:38.854134 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.853930 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-home\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-home\") pod \"3e57b4d8-7b73-4490-9138-de9401d6fd65\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " Oct 31 00:01:38.854134 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.853957 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e57b4d8-7b73-4490-9138-de9401d6fd65-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e57b4d8-7b73-4490-9138-de9401d6fd65-tls-certs\") pod \"3e57b4d8-7b73-4490-9138-de9401d6fd65\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " Oct 31 00:01:38.854134 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.853990 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-home\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-home\") pod \"42fdc936-a982-454f-a538-0a6494d1cfa2\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " Oct 31 00:01:38.854134 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854028 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-tmp-dir\") pod \"42fdc936-a982-454f-a538-0a6494d1cfa2\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " Oct 31 00:01:38.854134 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854074 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-home\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-home\") pod \"08cd1e28-c399-4cba-951e-0210a978109f\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " Oct 31 00:01:38.854134 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854100 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fdc936-a982-454f-a538-0a6494d1cfa2-kube-api-access-6mp2j\" (UniqueName: \"kubernetes.io/projected/42fdc936-a982-454f-a538-0a6494d1cfa2-kube-api-access-6mp2j\") pod \"42fdc936-a982-454f-a538-0a6494d1cfa2\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " Oct 31 00:01:38.854424 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854161 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-tmp-dir\") pod \"08cd1e28-c399-4cba-951e-0210a978109f\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " Oct 31 00:01:38.854424 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854187 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-dshm\") pod \"08cd1e28-c399-4cba-951e-0210a978109f\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " Oct 31 00:01:38.854424 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854240 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-model-cache\") pod \"42fdc936-a982-454f-a538-0a6494d1cfa2\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " Oct 31 00:01:38.854424 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854278 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fdc936-a982-454f-a538-0a6494d1cfa2-tls-certs\" (UniqueName: \"kubernetes.io/secret/42fdc936-a982-454f-a538-0a6494d1cfa2-tls-certs\") pod \"42fdc936-a982-454f-a538-0a6494d1cfa2\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " Oct 31 00:01:38.854424 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854332 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08cd1e28-c399-4cba-951e-0210a978109f-tls-certs\" (UniqueName: \"kubernetes.io/secret/08cd1e28-c399-4cba-951e-0210a978109f-tls-certs\") pod \"08cd1e28-c399-4cba-951e-0210a978109f\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " Oct 31 00:01:38.854424 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854402 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e57b4d8-7b73-4490-9138-de9401d6fd65-kube-api-access-x9smp\" (UniqueName: \"kubernetes.io/projected/3e57b4d8-7b73-4490-9138-de9401d6fd65-kube-api-access-x9smp\") pod \"3e57b4d8-7b73-4490-9138-de9401d6fd65\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " Oct 31 00:01:38.854759 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854497 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-model-cache\") pod \"08cd1e28-c399-4cba-951e-0210a978109f\" (UID: \"08cd1e28-c399-4cba-951e-0210a978109f\") " Oct 31 00:01:38.854759 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854571 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-model-cache\") pod \"3e57b4d8-7b73-4490-9138-de9401d6fd65\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " Oct 31 00:01:38.854759 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854622 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-tmp-dir\") pod \"3e57b4d8-7b73-4490-9138-de9401d6fd65\" (UID: \"3e57b4d8-7b73-4490-9138-de9401d6fd65\") " Oct 31 00:01:38.854759 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.854654 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-dshm\") pod \"42fdc936-a982-454f-a538-0a6494d1cfa2\" (UID: \"42fdc936-a982-454f-a538-0a6494d1cfa2\") " Oct 31 00:01:38.855223 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.855192 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-home" pod "3e57b4d8-7b73-4490-9138-de9401d6fd65" (UID: "3e57b4d8-7b73-4490-9138-de9401d6fd65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:38.855749 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.855451 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-model-cache" pod "08cd1e28-c399-4cba-951e-0210a978109f" (UID: "08cd1e28-c399-4cba-951e-0210a978109f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:38.856722 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.856698 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-home" pod "08cd1e28-c399-4cba-951e-0210a978109f" (UID: "08cd1e28-c399-4cba-951e-0210a978109f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:38.856952 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.856929 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-home" pod "42fdc936-a982-454f-a538-0a6494d1cfa2" (UID: "42fdc936-a982-454f-a538-0a6494d1cfa2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:38.857173 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.857151 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-tmp-dir" pod "42fdc936-a982-454f-a538-0a6494d1cfa2" (UID: "42fdc936-a982-454f-a538-0a6494d1cfa2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:38.857248 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.857190 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-tmp-dir" pod "08cd1e28-c399-4cba-951e-0210a978109f" (UID: "08cd1e28-c399-4cba-951e-0210a978109f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:38.857582 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.857557 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-model-cache" pod "3e57b4d8-7b73-4490-9138-de9401d6fd65" (UID: "3e57b4d8-7b73-4490-9138-de9401d6fd65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:38.857806 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.857783 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-tmp-dir" pod "3e57b4d8-7b73-4490-9138-de9401d6fd65" (UID: "3e57b4d8-7b73-4490-9138-de9401d6fd65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:38.858011 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.857989 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-model-cache" pod "42fdc936-a982-454f-a538-0a6494d1cfa2" (UID: "42fdc936-a982-454f-a538-0a6494d1cfa2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:38.858347 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.858308 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-dshm" pod "3e57b4d8-7b73-4490-9138-de9401d6fd65" (UID: "3e57b4d8-7b73-4490-9138-de9401d6fd65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:38.858907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.858882 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08cd1e28-c399-4cba-951e-0210a978109f-tls-certs" pod "08cd1e28-c399-4cba-951e-0210a978109f" (UID: "08cd1e28-c399-4cba-951e-0210a978109f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:38.860641 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.860590 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e57b4d8-7b73-4490-9138-de9401d6fd65-kube-api-access-x9smp" pod "3e57b4d8-7b73-4490-9138-de9401d6fd65" (UID: "3e57b4d8-7b73-4490-9138-de9401d6fd65"). InnerVolumeSpecName "kube-api-access-x9smp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:38.860641 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.860627 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fdc936-a982-454f-a538-0a6494d1cfa2-tls-certs" pod "42fdc936-a982-454f-a538-0a6494d1cfa2" (UID: "42fdc936-a982-454f-a538-0a6494d1cfa2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:38.862432 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.862407 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-dshm" pod "08cd1e28-c399-4cba-951e-0210a978109f" (UID: "08cd1e28-c399-4cba-951e-0210a978109f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:38.863625 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.862653 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e57b4d8-7b73-4490-9138-de9401d6fd65-tls-certs" pod "3e57b4d8-7b73-4490-9138-de9401d6fd65" (UID: "3e57b4d8-7b73-4490-9138-de9401d6fd65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:38.863625 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.862749 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cd1e28-c399-4cba-951e-0210a978109f-kube-api-access-cklbs" pod "08cd1e28-c399-4cba-951e-0210a978109f" (UID: "08cd1e28-c399-4cba-951e-0210a978109f"). InnerVolumeSpecName "kube-api-access-cklbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:38.863625 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.863129 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fdc936-a982-454f-a538-0a6494d1cfa2-kube-api-access-6mp2j" pod "42fdc936-a982-454f-a538-0a6494d1cfa2" (UID: "42fdc936-a982-454f-a538-0a6494d1cfa2"). InnerVolumeSpecName "kube-api-access-6mp2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:38.863625 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.863381 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-dshm" pod "42fdc936-a982-454f-a538-0a6494d1cfa2" (UID: "42fdc936-a982-454f-a538-0a6494d1cfa2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:38.894720 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.894689 7570 generic.go:356] "Generic (PLEG): container finished" podID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerID="f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2" exitCode=0 Oct 31 00:01:38.894882 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.894756 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" event={"ID":"3e57b4d8-7b73-4490-9138-de9401d6fd65","Type":"ContainerDied","Data":"f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2"} Oct 31 00:01:38.894882 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.894802 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" event={"ID":"3e57b4d8-7b73-4490-9138-de9401d6fd65","Type":"ContainerDied","Data":"12fc255c3bb7ece698a6f95273d1f617785a9d1dd3e15b0bbc2f9250f15c0550"} Oct 31 00:01:38.894882 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.894810 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl" Oct 31 00:01:38.894882 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.894823 7570 scope.go:122] "RemoveContainer" containerID="f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2" Oct 31 00:01:38.896626 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.896540 7570 generic.go:356] "Generic (PLEG): container finished" podID="08cd1e28-c399-4cba-951e-0210a978109f" containerID="46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5" exitCode=0 Oct 31 00:01:38.896626 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.896608 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" Oct 31 00:01:38.896805 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.896678 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" event={"ID":"08cd1e28-c399-4cba-951e-0210a978109f","Type":"ContainerDied","Data":"46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5"} Oct 31 00:01:38.896805 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.896699 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4" event={"ID":"08cd1e28-c399-4cba-951e-0210a978109f","Type":"ContainerDied","Data":"edb5c8fc6341300547078829125541cec8424c3d470ed622d86410a12c92d45a"} Oct 31 00:01:38.898605 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.898583 7570 generic.go:356] "Generic (PLEG): container finished" podID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerID="55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c" exitCode=0 Oct 31 00:01:38.898725 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.898655 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" event={"ID":"42fdc936-a982-454f-a538-0a6494d1cfa2","Type":"ContainerDied","Data":"55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c"} Oct 31 00:01:38.898725 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.898679 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" event={"ID":"42fdc936-a982-454f-a538-0a6494d1cfa2","Type":"ContainerDied","Data":"80f1d46443329c124eb024fe286a97f36d54b646c06b01a78aa955a1e810d63c"} Oct 31 00:01:38.898841 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.898733 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb" Oct 31 00:01:38.907953 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.907933 7570 scope.go:122] "RemoveContainer" containerID="f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2" Oct 31 00:01:38.908326 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:38.908303 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2\": container with ID starting with f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2 not found: ID does not exist" containerID="f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2" Oct 31 00:01:38.908397 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.908338 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2"} err="failed to get container status \"f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2\": rpc error: code = NotFound desc = could not find container \"f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2\": container with ID starting with f4411b60a3c4fadb0fd5dd99d7f69b53322b3bc63c1c71e45cb9b2d9c22320a2 not found: ID does not exist" Oct 31 00:01:38.908397 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.908362 7570 scope.go:122] "RemoveContainer" containerID="46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5" Oct 31 00:01:38.921656 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.921637 7570 scope.go:122] "RemoveContainer" containerID="46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5" Oct 31 00:01:38.922191 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:38.922176 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5\": container with ID starting with 46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5 not found: ID does not exist" containerID="46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5" Oct 31 00:01:38.922237 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.922198 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5"} err="failed to get container status \"46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5\": rpc error: code = NotFound desc = could not find container \"46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5\": container with ID starting with 46d1a3e85b5afedcbbb7030c521071f8fae17021d03e44e32370b565b0e1e2d5 not found: ID does not exist" Oct 31 00:01:38.922237 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.922213 7570 scope.go:122] "RemoveContainer" containerID="55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c" Oct 31 00:01:38.924187 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.924169 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl"] Oct 31 00:01:38.931612 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.931589 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/premium-w0-d3dcca-kserve-64955fd666-hx4zl"] Oct 31 00:01:38.933121 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.933101 7570 scope.go:122] "RemoveContainer" containerID="55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c" Oct 31 00:01:38.933388 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:38.933369 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c\": container with ID starting with 55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c not found: ID does not exist" containerID="55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c" Oct 31 00:01:38.933431 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.933395 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c"} err="failed to get container status \"55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c\": rpc error: code = NotFound desc = could not find container \"55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c\": container with ID starting with 55bb8a78cc3b0a8764a1cb1e248f6cf51b6d08691f41ce81f4c8e33312aa865c not found: ID does not exist" Oct 31 00:01:38.942913 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.942894 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4"] Oct 31 00:01:38.946889 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.946868 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/simulator-w0-d3dcca-kserve-74cd68d695-9s4b4"] Oct 31 00:01:38.956240 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956206 7570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-tmp-dir\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956240 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956232 7570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-dshm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956246 7570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-model-cache\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956260 7570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42fdc936-a982-454f-a538-0a6494d1cfa2-tls-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956274 7570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08cd1e28-c399-4cba-951e-0210a978109f-tls-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956287 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9smp\" (UniqueName: \"kubernetes.io/projected/3e57b4d8-7b73-4490-9138-de9401d6fd65-kube-api-access-x9smp\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956300 7570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-model-cache\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956313 7570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-model-cache\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956328 7570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-tmp-dir\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956340 7570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-dshm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956352 7570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-dshm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956364 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cklbs\" (UniqueName: \"kubernetes.io/projected/08cd1e28-c399-4cba-951e-0210a978109f-kube-api-access-cklbs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956378 7570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e57b4d8-7b73-4490-9138-de9401d6fd65-home\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956393 7570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e57b4d8-7b73-4490-9138-de9401d6fd65-tls-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956405 7570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-home\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.956423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956417 7570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fdc936-a982-454f-a538-0a6494d1cfa2-tmp-dir\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.957060 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956430 7570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08cd1e28-c399-4cba-951e-0210a978109f-home\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.957060 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.956443 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mp2j\" (UniqueName: \"kubernetes.io/projected/42fdc936-a982-454f-a538-0a6494d1cfa2-kube-api-access-6mp2j\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:38.973025 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.969142 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb"] Oct 31 00:01:38.982347 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:38.982272 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d3dcca/distinct-2-w0-d3dcca-kserve-bb9b5d6db-4hmhb"] Oct 31 00:01:40.602113 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:40.602084 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08cd1e28-c399-4cba-951e-0210a978109f" path="/var/lib/kubelet/pods/08cd1e28-c399-4cba-951e-0210a978109f/volumes" Oct 31 00:01:40.602554 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:40.602540 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" path="/var/lib/kubelet/pods/3e57b4d8-7b73-4490-9138-de9401d6fd65/volumes" Oct 31 00:01:40.602924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:40.602913 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" path="/var/lib/kubelet/pods/42fdc936-a982-454f-a538-0a6494d1cfa2/volumes" Oct 31 00:01:42.400331 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.400266 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" containerID="cri-o://a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1" gracePeriod=45 Oct 31 00:01:42.678260 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.678235 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 31 00:01:42.812371 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.812333 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tmp-dir\") pod \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " Oct 31 00:01:42.812371 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.812375 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-dshm\") pod \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " Oct 31 00:01:42.812693 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.812567 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-home\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-home\") pod \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " Oct 31 00:01:42.812693 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.812627 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tls-certs\" (UniqueName: \"kubernetes.io/secret/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tls-certs\") pod \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " Oct 31 00:01:42.812693 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.812662 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-model-cache\") pod \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " Oct 31 00:01:42.812835 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.812708 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tmp-dir" pod "223c27af-605a-42a8-a29c-ae1fa2b2ace6" (UID: "223c27af-605a-42a8-a29c-ae1fa2b2ace6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:42.812835 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.812800 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/223c27af-605a-42a8-a29c-ae1fa2b2ace6-kube-api-access-7kpqz\" (UniqueName: \"kubernetes.io/projected/223c27af-605a-42a8-a29c-ae1fa2b2ace6-kube-api-access-7kpqz\") pod \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\" (UID: \"223c27af-605a-42a8-a29c-ae1fa2b2ace6\") " Oct 31 00:01:42.812940 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.812884 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-home" pod "223c27af-605a-42a8-a29c-ae1fa2b2ace6" (UID: "223c27af-605a-42a8-a29c-ae1fa2b2ace6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:42.812993 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.812949 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-model-cache" pod "223c27af-605a-42a8-a29c-ae1fa2b2ace6" (UID: "223c27af-605a-42a8-a29c-ae1fa2b2ace6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:42.813241 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.813218 7570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-home\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:42.813241 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.813241 7570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-model-cache\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:42.813371 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.813255 7570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tmp-dir\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:42.814967 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.814942 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-dshm" pod "223c27af-605a-42a8-a29c-ae1fa2b2ace6" (UID: "223c27af-605a-42a8-a29c-ae1fa2b2ace6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:42.815057 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.815032 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223c27af-605a-42a8-a29c-ae1fa2b2ace6-kube-api-access-7kpqz" pod "223c27af-605a-42a8-a29c-ae1fa2b2ace6" (UID: "223c27af-605a-42a8-a29c-ae1fa2b2ace6"). InnerVolumeSpecName "kube-api-access-7kpqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:42.815355 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.815340 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tls-certs" pod "223c27af-605a-42a8-a29c-ae1fa2b2ace6" (UID: "223c27af-605a-42a8-a29c-ae1fa2b2ace6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:42.914778 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.914691 7570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223c27af-605a-42a8-a29c-ae1fa2b2ace6-tls-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:42.914778 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.914724 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kpqz\" (UniqueName: \"kubernetes.io/projected/223c27af-605a-42a8-a29c-ae1fa2b2ace6-kube-api-access-7kpqz\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:42.914778 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.914733 7570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223c27af-605a-42a8-a29c-ae1fa2b2ace6-dshm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:42.933356 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.933320 7570 generic.go:356] "Generic (PLEG): container finished" podID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerID="a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1" exitCode=0 Oct 31 00:01:42.933568 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.933373 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" Oct 31 00:01:42.933568 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.933404 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" event={"ID":"223c27af-605a-42a8-a29c-ae1fa2b2ace6","Type":"ContainerDied","Data":"a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1"} Oct 31 00:01:42.933568 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.933451 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs" event={"ID":"223c27af-605a-42a8-a29c-ae1fa2b2ace6","Type":"ContainerDied","Data":"bab552a50f2dfdd15ad55596a1c6215f8fd896fe4915884b568c766e72c1bf41"} Oct 31 00:01:42.933711 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.933518 7570 scope.go:122] "RemoveContainer" containerID="a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1" Oct 31 00:01:42.946389 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.946361 7570 scope.go:122] "RemoveContainer" containerID="a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1" Oct 31 00:01:42.946758 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:42.946737 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1\": container with ID starting with a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1 not found: ID does not exist" containerID="a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1" Oct 31 00:01:42.946826 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.946773 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1"} err="failed to get container status \"a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1\": rpc error: code = NotFound desc = could not find container \"a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1\": container with ID starting with a93e8921c6439f9cdc80889e9a5fdb0662688a4e0d422a5f352c9b01a22639c1 not found: ID does not exist" Oct 31 00:01:42.963717 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.963688 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs"] Oct 31 00:01:42.972380 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:42.972337 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/simulator-w2-9ef03a-kserve-cdb8584b5-76gjs"] Oct 31 00:01:43.154524 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:43.154456 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" podUID="1e8a6ea5-1c91-462f-94fd-7136a48971e9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:01:43.195057 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:43.194967 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:01:44.603341 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:44.603308 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" path="/var/lib/kubelet/pods/223c27af-605a-42a8-a29c-ae1fa2b2ace6/volumes" Oct 31 00:01:49.280865 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:49.280826 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8"] Oct 31 00:01:49.284055 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:49.284027 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d3dcca-szkc8"] Oct 31 00:01:49.961555 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:49.961519 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z"] Oct 31 00:01:49.961805 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:49.961768 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" containerID="cri-o://225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72" gracePeriod=30 Oct 31 00:01:50.269878 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.269853 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 31 00:01:50.306347 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.306294 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb"] Oct 31 00:01:50.312926 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.312899 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ef03a-2hdcb"] Oct 31 00:01:50.323343 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.323317 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15181ee-3e9a-40c9-a029-9ee504a971fa-kube-api-access-5xggl\" (UniqueName: \"kubernetes.io/projected/c15181ee-3e9a-40c9-a029-9ee504a971fa-kube-api-access-5xggl\") pod \"c15181ee-3e9a-40c9-a029-9ee504a971fa\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " Oct 31 00:01:50.323495 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.323405 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15181ee-3e9a-40c9-a029-9ee504a971fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15181ee-3e9a-40c9-a029-9ee504a971fa-openshift-service-ca\") pod \"c15181ee-3e9a-40c9-a029-9ee504a971fa\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " Oct 31 00:01:50.323571 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.323493 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-metrics-certs\") pod \"c15181ee-3e9a-40c9-a029-9ee504a971fa\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " Oct 31 00:01:50.323693 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.323676 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-maas-api-tls\") pod \"c15181ee-3e9a-40c9-a029-9ee504a971fa\" (UID: \"c15181ee-3e9a-40c9-a029-9ee504a971fa\") " Oct 31 00:01:50.323841 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.323812 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15181ee-3e9a-40c9-a029-9ee504a971fa-openshift-service-ca" pod "c15181ee-3e9a-40c9-a029-9ee504a971fa" (UID: "c15181ee-3e9a-40c9-a029-9ee504a971fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:50.324205 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.324155 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15181ee-3e9a-40c9-a029-9ee504a971fa-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:50.325943 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.325909 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15181ee-3e9a-40c9-a029-9ee504a971fa-kube-api-access-5xggl" pod "c15181ee-3e9a-40c9-a029-9ee504a971fa" (UID: "c15181ee-3e9a-40c9-a029-9ee504a971fa"). InnerVolumeSpecName "kube-api-access-5xggl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:50.326482 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.326430 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-maas-api-tls" pod "c15181ee-3e9a-40c9-a029-9ee504a971fa" (UID: "c15181ee-3e9a-40c9-a029-9ee504a971fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:50.326557 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.326538 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-metrics-certs" pod "c15181ee-3e9a-40c9-a029-9ee504a971fa" (UID: "c15181ee-3e9a-40c9-a029-9ee504a971fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:01:50.425687 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.425646 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:50.425687 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.425687 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15181ee-3e9a-40c9-a029-9ee504a971fa-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:50.427795 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.425706 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xggl\" (UniqueName: \"kubernetes.io/projected/c15181ee-3e9a-40c9-a029-9ee504a971fa-kube-api-access-5xggl\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:50.602396 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.602367 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ce6838f-955e-4414-8815-96fdf8880aec" path="/var/lib/kubelet/pods/4ce6838f-955e-4414-8815-96fdf8880aec/volumes" Oct 31 00:01:50.602821 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:50.602807 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95338565-81e8-4345-ba7a-d74ad08694a8" path="/var/lib/kubelet/pods/95338565-81e8-4345-ba7a-d74ad08694a8/volumes" Oct 31 00:01:51.008433 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:51.008351 7570 generic.go:356] "Generic (PLEG): container finished" podID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerID="225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72" exitCode=0 Oct 31 00:01:51.008433 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:51.008399 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" event={"ID":"c15181ee-3e9a-40c9-a029-9ee504a971fa","Type":"ContainerDied","Data":"225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72"} Oct 31 00:01:51.008629 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:51.008437 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" event={"ID":"c15181ee-3e9a-40c9-a029-9ee504a971fa","Type":"ContainerDied","Data":"043ba69bac9eb513802acf2f68d765dd15d3a300b711c95a5953e2f3739941a0"} Oct 31 00:01:51.008629 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:51.008400 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z" Oct 31 00:01:51.008629 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:51.008493 7570 scope.go:122] "RemoveContainer" containerID="225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72" Oct 31 00:01:51.019146 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:51.019128 7570 scope.go:122] "RemoveContainer" containerID="3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b" Oct 31 00:01:51.029429 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:51.029412 7570 scope.go:122] "RemoveContainer" containerID="225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72" Oct 31 00:01:51.029545 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:51.029525 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z"] Oct 31 00:01:51.029750 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:51.029730 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72\": container with ID starting with 225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72 not found: ID does not exist" containerID="225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72" Oct 31 00:01:51.029795 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:51.029759 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72"} err="failed to get container status \"225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72\": rpc error: code = NotFound desc = could not find container \"225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72\": container with ID starting with 225893b021753bb2d39d0d3bb73e81eda1c3be9cc5aafada72dc52d6e4a64f72 not found: ID does not exist" Oct 31 00:01:51.029795 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:51.029776 7570 scope.go:122] "RemoveContainer" containerID="3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b" Oct 31 00:01:51.030029 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:51.030012 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b\": container with ID starting with 3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b not found: ID does not exist" containerID="3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b" Oct 31 00:01:51.030092 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:51.030035 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b"} err="failed to get container status \"3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b\": rpc error: code = NotFound desc = could not find container \"3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b\": container with ID starting with 3a0c1e543fb08694a7fa8128fbf285148fdc52c0d068efa59bd4b73c20bda07b not found: ID does not exist" Oct 31 00:01:51.035415 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:51.035394 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ef03a-7c9878667-kjc6z"] Oct 31 00:01:52.602313 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:52.602281 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" path="/var/lib/kubelet/pods/c15181ee-3e9a-40c9-a029-9ee504a971fa/volumes" Oct 31 00:01:53.154501 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:53.154438 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" podUID="1e8a6ea5-1c91-462f-94fd-7136a48971e9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:01:53.194559 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:53.194522 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:01:54.051786 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.051520 7570 generic.go:356] "Generic (PLEG): container finished" podID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerID="c43dd4085f8264bf875fd779a96aad3200de0e9c03ed69d1979990a51aa50960" exitCode=0 Oct 31 00:01:54.051786 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.051716 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" event={"ID":"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3","Type":"ContainerDied","Data":"c43dd4085f8264bf875fd779a96aad3200de0e9c03ed69d1979990a51aa50960"} Oct 31 00:01:54.054799 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.054776 7570 generic.go:356] "Generic (PLEG): container finished" podID="1e8a6ea5-1c91-462f-94fd-7136a48971e9" containerID="cc6af65b1d32bcfd65a99cdbd83d31bf7073c45b98d943e624c0632032ffcd9f" exitCode=0 Oct 31 00:01:54.055149 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.055008 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" event={"ID":"1e8a6ea5-1c91-462f-94fd-7136a48971e9","Type":"ContainerDied","Data":"cc6af65b1d32bcfd65a99cdbd83d31bf7073c45b98d943e624c0632032ffcd9f"} Oct 31 00:01:54.096644 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.096620 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 31 00:01:54.155342 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.155315 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 31 00:01:54.174515 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.174488 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-plugins-config-volume\") pod \"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3\" (UID: \"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3\") " Oct 31 00:01:54.175027 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.175004 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-kube-api-access-s6vx6\" (UniqueName: \"kubernetes.io/projected/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-kube-api-access-s6vx6\") pod \"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3\" (UID: \"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3\") " Oct 31 00:01:54.175199 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.175173 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e8a6ea5-1c91-462f-94fd-7136a48971e9-kube-api-access-n4xqq\" (UniqueName: \"kubernetes.io/projected/1e8a6ea5-1c91-462f-94fd-7136a48971e9-kube-api-access-n4xqq\") pod \"1e8a6ea5-1c91-462f-94fd-7136a48971e9\" (UID: \"1e8a6ea5-1c91-462f-94fd-7136a48971e9\") " Oct 31 00:01:54.175317 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.175237 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8a6ea5-1c91-462f-94fd-7136a48971e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a6ea5-1c91-462f-94fd-7136a48971e9-plugins-config-volume\") pod \"1e8a6ea5-1c91-462f-94fd-7136a48971e9\" (UID: \"1e8a6ea5-1c91-462f-94fd-7136a48971e9\") " Oct 31 00:01:54.175401 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.175116 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-plugins-config-volume" pod "bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" (UID: "bfa36aaf-a967-456c-a544-9a2bcc4ac7d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:54.175771 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.175753 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:54.175957 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.175920 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8a6ea5-1c91-462f-94fd-7136a48971e9-plugins-config-volume" pod "1e8a6ea5-1c91-462f-94fd-7136a48971e9" (UID: "1e8a6ea5-1c91-462f-94fd-7136a48971e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:54.177920 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.177894 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-kube-api-access-s6vx6" pod "bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" (UID: "bfa36aaf-a967-456c-a544-9a2bcc4ac7d3"). InnerVolumeSpecName "kube-api-access-s6vx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:54.178135 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.178107 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8a6ea5-1c91-462f-94fd-7136a48971e9-kube-api-access-n4xqq" pod "1e8a6ea5-1c91-462f-94fd-7136a48971e9" (UID: "1e8a6ea5-1c91-462f-94fd-7136a48971e9"). InnerVolumeSpecName "kube-api-access-n4xqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:54.277204 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.277126 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6vx6\" (UniqueName: \"kubernetes.io/projected/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3-kube-api-access-s6vx6\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:54.277204 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.277154 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4xqq\" (UniqueName: \"kubernetes.io/projected/1e8a6ea5-1c91-462f-94fd-7136a48971e9-kube-api-access-n4xqq\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:54.277204 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:54.277163 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a6ea5-1c91-462f-94fd-7136a48971e9-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:55.062878 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:55.062784 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" Oct 31 00:01:55.062878 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:55.062816 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm" event={"ID":"1e8a6ea5-1c91-462f-94fd-7136a48971e9","Type":"ContainerDied","Data":"9d936ff83b731740c54fdf7e84d5ccb4049f748048aaeff41a588dc5b33e4a6b"} Oct 31 00:01:55.063550 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:55.062885 7570 scope.go:122] "RemoveContainer" containerID="cc6af65b1d32bcfd65a99cdbd83d31bf7073c45b98d943e624c0632032ffcd9f" Oct 31 00:01:55.064704 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:55.064686 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" Oct 31 00:01:55.064817 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:55.064689 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb" event={"ID":"bfa36aaf-a967-456c-a544-9a2bcc4ac7d3","Type":"ContainerDied","Data":"0d2b0c3363576a16525c48f9615c79028ff46d82f4bc3276437a0d5e9c5a2d0e"} Oct 31 00:01:55.074271 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:55.074252 7570 scope.go:122] "RemoveContainer" containerID="c43dd4085f8264bf875fd779a96aad3200de0e9c03ed69d1979990a51aa50960" Oct 31 00:01:55.086315 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:55.086289 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm"] Oct 31 00:01:55.092637 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:55.092609 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d3dcca-549fbc74b7-dq2jm"] Oct 31 00:01:55.107813 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:55.107788 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb"] Oct 31 00:01:55.112042 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:55.112020 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d3dcca-549c8d9767-jbrzb"] Oct 31 00:01:56.601840 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:56.601804 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e8a6ea5-1c91-462f-94fd-7136a48971e9" path="/var/lib/kubelet/pods/1e8a6ea5-1c91-462f-94fd-7136a48971e9/volumes" Oct 31 00:01:56.602923 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:56.602907 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" path="/var/lib/kubelet/pods/bfa36aaf-a967-456c-a544-9a2bcc4ac7d3/volumes" Oct 31 00:01:57.102283 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:57.102246 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b"] Oct 31 00:01:57.102529 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:57.102503 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" containerID="cri-o://29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b" gracePeriod=30 Oct 31 00:01:57.106326 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:57.106301 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l"] Oct 31 00:01:57.106540 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:57.106518 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" containerID="cri-o://50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e" gracePeriod=30 Oct 31 00:01:58.130763 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:58.130717 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:01:58.131261 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:58.130776 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:01:58.131261 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:58.130967 7570 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:01:58.131261 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:58.131014 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:01:58.392185 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:58.392096 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv"] Oct 31 00:01:58.392382 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:58.392357 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" containerID="cri-o://2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614" gracePeriod=30 Oct 31 00:01:58.406588 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:58.406564 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724"] Oct 31 00:01:58.406767 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:01:58.406749 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" containerID="cri-o://47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e" gracePeriod=30 Oct 31 00:01:59.412578 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:01:59.412538 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:02:02.427716 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.427688 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 31 00:02:02.431797 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.431772 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 31 00:02:02.586887 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.586833 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istiod-ca-cert\") pod \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " Oct 31 00:02:02.587094 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.586900 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-kube-api-access-qswdq\" (UniqueName: \"kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-kube-api-access-qswdq\") pod \"cf77418c-72fd-40ad-b389-df9273f36cfc\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " Oct 31 00:02:02.587094 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.586939 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-podinfo\") pod \"cf77418c-72fd-40ad-b389-df9273f36cfc\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " Oct 31 00:02:02.587094 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.586987 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-envoy\") pod \"cf77418c-72fd-40ad-b389-df9273f36cfc\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " Oct 31 00:02:02.587094 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587067 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-envoy\") pod \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " Oct 31 00:02:02.587325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587125 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-data\") pod \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " Oct 31 00:02:02.587325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587164 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf77418c-72fd-40ad-b389-df9273f36cfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf77418c-72fd-40ad-b389-df9273f36cfc-istiod-ca-cert\") pod \"cf77418c-72fd-40ad-b389-df9273f36cfc\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " Oct 31 00:02:02.587325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587194 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-certs\") pod \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " Oct 31 00:02:02.587325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587223 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-podinfo\") pod \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " Oct 31 00:02:02.587325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587267 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-kube-api-access-wb5qh\" (UniqueName: \"kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-kube-api-access-wb5qh\") pod \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " Oct 31 00:02:02.587325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587317 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-token\" (UniqueName: \"kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-token\") pod \"cf77418c-72fd-40ad-b389-df9273f36cfc\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " Oct 31 00:02:02.587653 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587347 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-credential-socket\") pod \"cf77418c-72fd-40ad-b389-df9273f36cfc\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " Oct 31 00:02:02.587653 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587378 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-socket\") pod \"cf77418c-72fd-40ad-b389-df9273f36cfc\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " Oct 31 00:02:02.587653 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587425 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-socket\") pod \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " Oct 31 00:02:02.587653 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587498 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-token\" (UniqueName: \"kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-token\") pod \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " Oct 31 00:02:02.587653 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587596 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-certs\") pod \"cf77418c-72fd-40ad-b389-df9273f36cfc\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " Oct 31 00:02:02.587653 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587630 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-data\") pod \"cf77418c-72fd-40ad-b389-df9273f36cfc\" (UID: \"cf77418c-72fd-40ad-b389-df9273f36cfc\") " Oct 31 00:02:02.587954 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.587663 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-credential-socket\") pod \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\" (UID: \"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f\") " Oct 31 00:02:02.589427 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.589390 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-data" pod "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" (UID: "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:02.590208 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.590018 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-socket" pod "cf77418c-72fd-40ad-b389-df9273f36cfc" (UID: "cf77418c-72fd-40ad-b389-df9273f36cfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:02.590305 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.590244 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-credential-socket" pod "cf77418c-72fd-40ad-b389-df9273f36cfc" (UID: "cf77418c-72fd-40ad-b389-df9273f36cfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:02.590691 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.590450 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-certs" pod "cf77418c-72fd-40ad-b389-df9273f36cfc" (UID: "cf77418c-72fd-40ad-b389-df9273f36cfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:02.590691 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.590508 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-socket" pod "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" (UID: "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:02.590843 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.590701 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-data" pod "cf77418c-72fd-40ad-b389-df9273f36cfc" (UID: "cf77418c-72fd-40ad-b389-df9273f36cfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:02.591149 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.591125 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-credential-socket" pod "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" (UID: "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:02.591435 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.591414 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf77418c-72fd-40ad-b389-df9273f36cfc-istiod-ca-cert" pod "cf77418c-72fd-40ad-b389-df9273f36cfc" (UID: "cf77418c-72fd-40ad-b389-df9273f36cfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:02.593382 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.593099 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-envoy" pod "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" (UID: "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:02.593382 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.593134 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-podinfo" pod "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" (UID: "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:02:02.593656 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.593562 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istiod-ca-cert" pod "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" (UID: "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:02.593893 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.593865 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-certs" pod "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" (UID: "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:02.595356 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.595231 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-podinfo" pod "cf77418c-72fd-40ad-b389-df9273f36cfc" (UID: "cf77418c-72fd-40ad-b389-df9273f36cfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:02:02.595356 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.595289 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-token" pod "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" (UID: "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:02.595532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.595367 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-envoy" pod "cf77418c-72fd-40ad-b389-df9273f36cfc" (UID: "cf77418c-72fd-40ad-b389-df9273f36cfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:02.595532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.595397 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-token" pod "cf77418c-72fd-40ad-b389-df9273f36cfc" (UID: "cf77418c-72fd-40ad-b389-df9273f36cfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:02.595633 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.595596 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-kube-api-access-wb5qh" pod "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" (UID: "cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f"). InnerVolumeSpecName "kube-api-access-wb5qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:02.595757 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.595733 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-kube-api-access-qswdq" pod "cf77418c-72fd-40ad-b389-df9273f36cfc" (UID: "cf77418c-72fd-40ad-b389-df9273f36cfc"). InnerVolumeSpecName "kube-api-access-qswdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:02.690618 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690529 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690618 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690562 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690618 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690576 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690618 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690587 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690618 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690602 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qswdq\" (UniqueName: \"kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-kube-api-access-qswdq\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690618 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690615 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690629 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690641 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690653 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690663 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf77418c-72fd-40ad-b389-df9273f36cfc-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690673 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690684 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690696 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb5qh\" (UniqueName: \"kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-kube-api-access-wb5qh\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690707 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf77418c-72fd-40ad-b389-df9273f36cfc-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690721 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690735 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf77418c-72fd-40ad-b389-df9273f36cfc-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690750 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:02.690936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:02.690762 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.111192 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111161 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v"] Oct 31 00:02:03.111669 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111653 7570 cpu_manager.go:414] "containerMap: removing container" podUID="47465535-f16b-4864-a764-71f07c0d273d" containerName="revoke-keys" Oct 31 00:02:03.111724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111671 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47465535-f16b-4864-a764-71f07c0d273d" containerName="revoke-keys" Oct 31 00:02:03.111724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111686 7570 cpu_manager.go:414] "containerMap: removing container" podUID="1ab1e878-bca9-41c4-81e6-0001487f5ff1" containerName="revoke-keys" Oct 31 00:02:03.111724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111691 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab1e878-bca9-41c4-81e6-0001487f5ff1" containerName="revoke-keys" Oct 31 00:02:03.111724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111703 7570 cpu_manager.go:414] "containerMap: removing container" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" Oct 31 00:02:03.111724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111708 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" Oct 31 00:02:03.111724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111719 7570 cpu_manager.go:414] "containerMap: removing container" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" Oct 31 00:02:03.111724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111723 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111733 7570 cpu_manager.go:414] "containerMap: removing container" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerName="payload-processing" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111738 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerName="payload-processing" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111746 7570 cpu_manager.go:414] "containerMap: removing container" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111750 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111758 7570 cpu_manager.go:414] "containerMap: removing container" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111762 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111772 7570 cpu_manager.go:414] "containerMap: removing container" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111776 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111795 7570 cpu_manager.go:414] "containerMap: removing container" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111799 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111810 7570 cpu_manager.go:414] "containerMap: removing container" podUID="1e8a6ea5-1c91-462f-94fd-7136a48971e9" containerName="payload-pre-processing" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111814 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8a6ea5-1c91-462f-94fd-7136a48971e9" containerName="payload-pre-processing" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111822 7570 cpu_manager.go:414] "containerMap: removing container" podUID="28216477-7eed-4e96-a7e7-262ab1d60bb9" containerName="revoke-keys" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111826 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28216477-7eed-4e96-a7e7-262ab1d60bb9" containerName="revoke-keys" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111835 7570 cpu_manager.go:414] "containerMap: removing container" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111838 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111901 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="223c27af-605a-42a8-a29c-ae1fa2b2ace6" containerName="main" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111909 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerName="istio-proxy" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111917 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerName="istio-proxy" Oct 31 00:02:03.111924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111926 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e57b4d8-7b73-4490-9138-de9401d6fd65" containerName="main" Oct 31 00:02:03.112498 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111934 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8a6ea5-1c91-462f-94fd-7136a48971e9" containerName="payload-pre-processing" Oct 31 00:02:03.112498 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111940 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="28216477-7eed-4e96-a7e7-262ab1d60bb9" containerName="revoke-keys" Oct 31 00:02:03.112498 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111947 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" Oct 31 00:02:03.112498 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111951 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="08cd1e28-c399-4cba-951e-0210a978109f" containerName="main" Oct 31 00:02:03.112498 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111956 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="47465535-f16b-4864-a764-71f07c0d273d" containerName="revoke-keys" Oct 31 00:02:03.112498 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111962 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab1e878-bca9-41c4-81e6-0001487f5ff1" containerName="revoke-keys" Oct 31 00:02:03.112498 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111968 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fdc936-a982-454f-a538-0a6494d1cfa2" containerName="main" Oct 31 00:02:03.112498 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.111973 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa36aaf-a967-456c-a544-9a2bcc4ac7d3" containerName="payload-processing" Oct 31 00:02:03.116617 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.116602 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:03.121737 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.121708 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:02:03.121881 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.121825 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:02:03.121881 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.121873 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:02:03.121992 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.121926 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:02:03.127513 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.126684 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v"] Oct 31 00:02:03.141016 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.140988 7570 generic.go:356] "Generic (PLEG): container finished" podID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" containerID="29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b" exitCode=0 Oct 31 00:02:03.141560 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.141079 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" Oct 31 00:02:03.141560 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.141104 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" event={"ID":"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f","Type":"ContainerDied","Data":"29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b"} Oct 31 00:02:03.141560 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.141147 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b" event={"ID":"cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f","Type":"ContainerDied","Data":"5cd54dc16767b953c9edebbddcc553b00da872d7ec853df4ed83626ecfd5335f"} Oct 31 00:02:03.141560 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.141179 7570 scope.go:122] "RemoveContainer" containerID="29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b" Oct 31 00:02:03.146236 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.146206 7570 generic.go:356] "Generic (PLEG): container finished" podID="cf77418c-72fd-40ad-b389-df9273f36cfc" containerID="50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e" exitCode=0 Oct 31 00:02:03.146342 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.146287 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" Oct 31 00:02:03.146766 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.146287 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" event={"ID":"cf77418c-72fd-40ad-b389-df9273f36cfc","Type":"ContainerDied","Data":"50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e"} Oct 31 00:02:03.146854 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.146803 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l" event={"ID":"cf77418c-72fd-40ad-b389-df9273f36cfc","Type":"ContainerDied","Data":"56cfcfa4e3a05425fd1c4bc22f832e661418b9535ed0d26de65b6ad8a4377f70"} Oct 31 00:02:03.163191 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.163169 7570 scope.go:122] "RemoveContainer" containerID="29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b" Oct 31 00:02:03.163612 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:02:03.163586 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b\": container with ID starting with 29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b not found: ID does not exist" containerID="29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b" Oct 31 00:02:03.163712 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.163622 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b"} err="failed to get container status \"29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b\": rpc error: code = NotFound desc = could not find container \"29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b\": container with ID starting with 29947cabd1f6ad2ef4e90fdb1438a8199c7634ac7d36076c791eb286714ba71b not found: ID does not exist" Oct 31 00:02:03.163712 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.163643 7570 scope.go:122] "RemoveContainer" containerID="50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e" Oct 31 00:02:03.173245 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.173219 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l"] Oct 31 00:02:03.176811 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.176789 7570 scope.go:122] "RemoveContainer" containerID="50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e" Oct 31 00:02:03.177190 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:02:03.177165 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e\": container with ID starting with 50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e not found: ID does not exist" containerID="50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e" Oct 31 00:02:03.177268 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.177201 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e"} err="failed to get container status \"50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e\": rpc error: code = NotFound desc = could not find container \"50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e\": container with ID starting with 50e39c3fca6e00e777faa7b2d9eba0d0fadd6ce733455b5dca6358c118637b9e not found: ID does not exist" Oct 31 00:02:03.177316 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.177279 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-j8l9l"] Oct 31 00:02:03.195327 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.195201 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b"] Oct 31 00:02:03.212179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.207331 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4g5p\" (UniqueName: \"kubernetes.io/projected/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-kube-api-access-g4g5p\") pod \"maas-api-revoke-sub-7e04d396-bkt7v\" (UID: \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:03.212179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.210231 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e04d396-bkt7v\" (UID: \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:03.212179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.210293 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e04d396-bkt7v\" (UID: \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:03.212179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.212019 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-52k4b"] Oct 31 00:02:03.311637 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.311607 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e04d396-bkt7v\" (UID: \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:03.311637 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.311650 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e04d396-bkt7v\" (UID: \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:03.311829 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.311738 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4g5p\" (UniqueName: \"kubernetes.io/projected/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-kube-api-access-g4g5p\") pod \"maas-api-revoke-sub-7e04d396-bkt7v\" (UID: \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:03.312238 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.312215 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e04d396-bkt7v\" (UID: \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:03.314183 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.314161 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e04d396-bkt7v\" (UID: \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:03.324210 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.324186 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4g5p\" (UniqueName: \"kubernetes.io/projected/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-kube-api-access-g4g5p\") pod \"maas-api-revoke-sub-7e04d396-bkt7v\" (UID: \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:03.459267 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.459231 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:03.673747 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.673675 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 31 00:02:03.741355 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.741330 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 31 00:02:03.755212 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.755170 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v"] Oct 31 00:02:03.756419 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:02:03.756391 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9ded63_bf41_43a1_8b50_33fadb1c57fe.slice/crio-87be9552f567127051c824f45dc00747d3cf983b2f0e6b67c291ebaa2c8ac037 WatchSource:0}: Error finding container 87be9552f567127051c824f45dc00747d3cf983b2f0e6b67c291ebaa2c8ac037: Status 404 returned error can't find the container with id 87be9552f567127051c824f45dc00747d3cf983b2f0e6b67c291ebaa2c8ac037 Oct 31 00:02:03.829622 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.829595 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-token\" (UniqueName: \"kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-token\") pod \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " Oct 31 00:02:03.829772 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.829639 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-data\") pod \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " Oct 31 00:02:03.829772 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.829752 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istiod-ca-cert\") pod \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " Oct 31 00:02:03.829850 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.829776 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-socket\") pod \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " Oct 31 00:02:03.829850 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.829811 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-kube-api-access-7rqhk\" (UniqueName: \"kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-kube-api-access-7rqhk\") pod \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " Oct 31 00:02:03.829850 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.829833 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-podinfo\") pod \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " Oct 31 00:02:03.829985 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.829878 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-credential-socket\") pod \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " Oct 31 00:02:03.829985 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.829897 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-envoy\") pod \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " Oct 31 00:02:03.829985 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.829923 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-certs\") pod \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\" (UID: \"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9\") " Oct 31 00:02:03.830128 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.830073 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-data" pod "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" (UID: "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:03.830358 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.830340 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.830696 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.830673 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istiod-ca-cert" pod "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" (UID: "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:03.830807 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.830701 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-socket" pod "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" (UID: "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:03.830896 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.830870 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-credential-socket" pod "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" (UID: "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:03.831316 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.831098 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-certs" pod "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" (UID: "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:03.833061 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.833031 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-podinfo" pod "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" (UID: "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:02:03.833061 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.833046 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-token" pod "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" (UID: "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:03.833520 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.833495 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-envoy" pod "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" (UID: "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:03.833596 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.833526 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-kube-api-access-7rqhk" pod "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" (UID: "4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9"). InnerVolumeSpecName "kube-api-access-7rqhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:03.930787 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.930726 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-kube-api-access-8sf6f\" (UniqueName: \"kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-kube-api-access-8sf6f\") pod \"d2658215-b85d-44d6-8d22-9803dd36e9f4\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " Oct 31 00:02:03.930787 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.930768 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-credential-socket\") pod \"d2658215-b85d-44d6-8d22-9803dd36e9f4\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " Oct 31 00:02:03.930948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.930823 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-podinfo\") pod \"d2658215-b85d-44d6-8d22-9803dd36e9f4\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " Oct 31 00:02:03.930948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.930866 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-socket\") pod \"d2658215-b85d-44d6-8d22-9803dd36e9f4\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " Oct 31 00:02:03.930948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.930888 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-data\") pod \"d2658215-b85d-44d6-8d22-9803dd36e9f4\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " Oct 31 00:02:03.930948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.930927 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-envoy\") pod \"d2658215-b85d-44d6-8d22-9803dd36e9f4\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " Oct 31 00:02:03.931135 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931005 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2658215-b85d-44d6-8d22-9803dd36e9f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2658215-b85d-44d6-8d22-9803dd36e9f4-istiod-ca-cert\") pod \"d2658215-b85d-44d6-8d22-9803dd36e9f4\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " Oct 31 00:02:03.931135 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931048 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-certs\") pod \"d2658215-b85d-44d6-8d22-9803dd36e9f4\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " Oct 31 00:02:03.931135 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931090 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-token\" (UniqueName: \"kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-token\") pod \"d2658215-b85d-44d6-8d22-9803dd36e9f4\" (UID: \"d2658215-b85d-44d6-8d22-9803dd36e9f4\") " Oct 31 00:02:03.931271 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931129 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-credential-socket" pod "d2658215-b85d-44d6-8d22-9803dd36e9f4" (UID: "d2658215-b85d-44d6-8d22-9803dd36e9f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:03.931271 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931221 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-socket" pod "d2658215-b85d-44d6-8d22-9803dd36e9f4" (UID: "d2658215-b85d-44d6-8d22-9803dd36e9f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:03.931386 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931264 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-data" pod "d2658215-b85d-44d6-8d22-9803dd36e9f4" (UID: "d2658215-b85d-44d6-8d22-9803dd36e9f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:03.931437 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931415 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-certs" pod "d2658215-b85d-44d6-8d22-9803dd36e9f4" (UID: "d2658215-b85d-44d6-8d22-9803dd36e9f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:03.931514 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931439 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2658215-b85d-44d6-8d22-9803dd36e9f4-istiod-ca-cert" pod "d2658215-b85d-44d6-8d22-9803dd36e9f4" (UID: "d2658215-b85d-44d6-8d22-9803dd36e9f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931815 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931843 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931860 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931874 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931888 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931903 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931917 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931932 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2658215-b85d-44d6-8d22-9803dd36e9f4-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931949 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931962 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931975 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.931989 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.931948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.932004 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rqhk\" (UniqueName: \"kubernetes.io/projected/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9-kube-api-access-7rqhk\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:03.933285 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.933265 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-podinfo" pod "d2658215-b85d-44d6-8d22-9803dd36e9f4" (UID: "d2658215-b85d-44d6-8d22-9803dd36e9f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:02:03.933361 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.933303 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-kube-api-access-8sf6f" pod "d2658215-b85d-44d6-8d22-9803dd36e9f4" (UID: "d2658215-b85d-44d6-8d22-9803dd36e9f4"). InnerVolumeSpecName "kube-api-access-8sf6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:03.933361 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.933319 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-envoy" pod "d2658215-b85d-44d6-8d22-9803dd36e9f4" (UID: "d2658215-b85d-44d6-8d22-9803dd36e9f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:03.933642 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:03.933627 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-token" pod "d2658215-b85d-44d6-8d22-9803dd36e9f4" (UID: "d2658215-b85d-44d6-8d22-9803dd36e9f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:04.032782 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.032757 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sf6f\" (UniqueName: \"kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-kube-api-access-8sf6f\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:04.032782 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.032780 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:04.032907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.032789 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:04.032907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.032797 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2658215-b85d-44d6-8d22-9803dd36e9f4-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:04.154232 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.154197 7570 generic.go:356] "Generic (PLEG): container finished" podID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerID="47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e" exitCode=0 Oct 31 00:02:04.154361 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.154268 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" Oct 31 00:02:04.154361 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.154271 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" event={"ID":"d2658215-b85d-44d6-8d22-9803dd36e9f4","Type":"ContainerDied","Data":"47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e"} Oct 31 00:02:04.154361 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.154340 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724" event={"ID":"d2658215-b85d-44d6-8d22-9803dd36e9f4","Type":"ContainerDied","Data":"7236e5d7c15f1aef222038b0f63cca98fac7bba5b2ee6e4726eea0d330d588ef"} Oct 31 00:02:04.154561 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.154371 7570 scope.go:122] "RemoveContainer" containerID="47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e" Oct 31 00:02:04.156214 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.156185 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" event={"ID":"1d9ded63-bf41-43a1-8b50-33fadb1c57fe","Type":"ContainerStarted","Data":"f8378e18462d840aa7fab94c24fbfad1907786111a3722693cf6639b044f0618"} Oct 31 00:02:04.156323 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.156235 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" event={"ID":"1d9ded63-bf41-43a1-8b50-33fadb1c57fe","Type":"ContainerStarted","Data":"87be9552f567127051c824f45dc00747d3cf983b2f0e6b67c291ebaa2c8ac037"} Oct 31 00:02:04.157654 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.157634 7570 generic.go:356] "Generic (PLEG): container finished" podID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerID="2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614" exitCode=0 Oct 31 00:02:04.157767 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.157695 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" Oct 31 00:02:04.157767 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.157716 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" event={"ID":"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9","Type":"ContainerDied","Data":"2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614"} Oct 31 00:02:04.157767 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.157758 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv" event={"ID":"4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9","Type":"ContainerDied","Data":"741e6d04777b0e73aa758c809f60a4ac99f48d5d680e568ff6f397cd9aff0231"} Oct 31 00:02:04.167119 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.167103 7570 scope.go:122] "RemoveContainer" containerID="47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e" Oct 31 00:02:04.167517 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:02:04.167483 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e\": container with ID starting with 47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e not found: ID does not exist" containerID="47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e" Oct 31 00:02:04.167618 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.167522 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e"} err="failed to get container status \"47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e\": rpc error: code = NotFound desc = could not find container \"47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e\": container with ID starting with 47ea4f97fc4cc450ce4d85e382f24849083c3a4db53589274ad16f3b3e2c1c5e not found: ID does not exist" Oct 31 00:02:04.167618 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.167544 7570 scope.go:122] "RemoveContainer" containerID="2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614" Oct 31 00:02:04.178102 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.178088 7570 scope.go:122] "RemoveContainer" containerID="2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614" Oct 31 00:02:04.178355 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:02:04.178338 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614\": container with ID starting with 2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614 not found: ID does not exist" containerID="2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614" Oct 31 00:02:04.178417 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.178362 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614"} err="failed to get container status \"2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614\": rpc error: code = NotFound desc = could not find container \"2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614\": container with ID starting with 2df5b80be5cb35294bc23203f457297b6a3f62c1c8652dd14cd024631b826614 not found: ID does not exist" Oct 31 00:02:04.182392 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.182328 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" podStartSLOduration=1.182317418 podStartE2EDuration="1.182317418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:04.18198941 +0000 UTC m=+2535.700990501" watchObservedRunningTime="2026-10-31 00:02:04.182317418 +0000 UTC m=+2535.701318507" Oct 31 00:02:04.200048 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.200023 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv"] Oct 31 00:02:04.209485 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.207952 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-rrqnv"] Oct 31 00:02:04.222435 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.222402 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724"] Oct 31 00:02:04.225697 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.225674 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-lm724"] Oct 31 00:02:04.601990 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.601961 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" path="/var/lib/kubelet/pods/4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9/volumes" Oct 31 00:02:04.602420 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.602407 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f" path="/var/lib/kubelet/pods/cdee70e9-f8d5-40e2-a4e4-bb2fc009db4f/volumes" Oct 31 00:02:04.602839 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.602827 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf77418c-72fd-40ad-b389-df9273f36cfc" path="/var/lib/kubelet/pods/cf77418c-72fd-40ad-b389-df9273f36cfc/volumes" Oct 31 00:02:04.603550 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:04.603536 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" path="/var/lib/kubelet/pods/d2658215-b85d-44d6-8d22-9803dd36e9f4/volumes" Oct 31 00:02:05.168910 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:05.168875 7570 generic.go:356] "Generic (PLEG): container finished" podID="1d9ded63-bf41-43a1-8b50-33fadb1c57fe" containerID="f8378e18462d840aa7fab94c24fbfad1907786111a3722693cf6639b044f0618" exitCode=0 Oct 31 00:02:05.169103 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:05.168964 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" event={"ID":"1d9ded63-bf41-43a1-8b50-33fadb1c57fe","Type":"ContainerDied","Data":"f8378e18462d840aa7fab94c24fbfad1907786111a3722693cf6639b044f0618"} Oct 31 00:02:06.321722 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:06.321698 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:06.358993 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:06.358964 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-xks-ca\") pod \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\" (UID: \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\") " Oct 31 00:02:06.359155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:06.359050 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-kube-api-access-g4g5p\" (UniqueName: \"kubernetes.io/projected/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-kube-api-access-g4g5p\") pod \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\" (UID: \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\") " Oct 31 00:02:06.359217 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:06.359201 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-openshift-ca\") pod \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\" (UID: \"1d9ded63-bf41-43a1-8b50-33fadb1c57fe\") " Oct 31 00:02:06.359644 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:06.359577 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-openshift-ca" pod "1d9ded63-bf41-43a1-8b50-33fadb1c57fe" (UID: "1d9ded63-bf41-43a1-8b50-33fadb1c57fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:06.361270 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:06.361242 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-xks-ca" pod "1d9ded63-bf41-43a1-8b50-33fadb1c57fe" (UID: "1d9ded63-bf41-43a1-8b50-33fadb1c57fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:06.361359 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:06.361305 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-kube-api-access-g4g5p" pod "1d9ded63-bf41-43a1-8b50-33fadb1c57fe" (UID: "1d9ded63-bf41-43a1-8b50-33fadb1c57fe"). InnerVolumeSpecName "kube-api-access-g4g5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:06.461269 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:06.461175 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:06.461269 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:06.461214 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:06.461269 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:06.461224 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4g5p\" (UniqueName: \"kubernetes.io/projected/1d9ded63-bf41-43a1-8b50-33fadb1c57fe-kube-api-access-g4g5p\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:07.183013 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:07.182982 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" Oct 31 00:02:07.183209 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:07.183027 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v" event={"ID":"1d9ded63-bf41-43a1-8b50-33fadb1c57fe","Type":"ContainerDied","Data":"87be9552f567127051c824f45dc00747d3cf983b2f0e6b67c291ebaa2c8ac037"} Oct 31 00:02:07.183209 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:07.183082 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87be9552f567127051c824f45dc00747d3cf983b2f0e6b67c291ebaa2c8ac037" Oct 31 00:02:14.137694 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.137617 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x"] Oct 31 00:02:14.138196 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.138179 7570 cpu_manager.go:414] "containerMap: removing container" podUID="1d9ded63-bf41-43a1-8b50-33fadb1c57fe" containerName="revoke-keys" Oct 31 00:02:14.138241 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.138199 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9ded63-bf41-43a1-8b50-33fadb1c57fe" containerName="revoke-keys" Oct 31 00:02:14.138241 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.138219 7570 cpu_manager.go:414] "containerMap: removing container" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" Oct 31 00:02:14.138241 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.138224 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" Oct 31 00:02:14.138336 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.138242 7570 cpu_manager.go:414] "containerMap: removing container" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" Oct 31 00:02:14.138336 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.138248 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" Oct 31 00:02:14.138336 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.138264 7570 cpu_manager.go:414] "containerMap: removing container" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" Oct 31 00:02:14.138336 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.138268 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" Oct 31 00:02:14.138446 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.138347 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15181ee-3e9a-40c9-a029-9ee504a971fa" containerName="maas-api" Oct 31 00:02:14.138446 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.138356 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2658215-b85d-44d6-8d22-9803dd36e9f4" containerName="istio-proxy" Oct 31 00:02:14.138446 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.138369 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9ded63-bf41-43a1-8b50-33fadb1c57fe" containerName="revoke-keys" Oct 31 00:02:14.138446 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.138377 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed6a3e1-fd2f-49b8-9af6-5d9a49f9d3e9" containerName="istio-proxy" Oct 31 00:02:14.142742 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.142728 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:14.148777 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.148747 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:02:14.148895 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.148789 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:02:14.152900 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.152880 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x"] Oct 31 00:02:14.207401 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.207365 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:02:14.207546 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.207375 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:02:14.252821 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.252794 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5aba07a-x585x\" (UID: \"6d36cc25-4820-4388-8e5d-a512071934d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:14.252821 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.252826 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grbxx\" (UniqueName: \"kubernetes.io/projected/6d36cc25-4820-4388-8e5d-a512071934d3-kube-api-access-grbxx\") pod \"maas-api-revoke-sub-d5aba07a-x585x\" (UID: \"6d36cc25-4820-4388-8e5d-a512071934d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:14.252994 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.252913 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5aba07a-x585x\" (UID: \"6d36cc25-4820-4388-8e5d-a512071934d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:14.354167 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.354133 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5aba07a-x585x\" (UID: \"6d36cc25-4820-4388-8e5d-a512071934d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:14.354167 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.354167 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grbxx\" (UniqueName: \"kubernetes.io/projected/6d36cc25-4820-4388-8e5d-a512071934d3-kube-api-access-grbxx\") pod \"maas-api-revoke-sub-d5aba07a-x585x\" (UID: \"6d36cc25-4820-4388-8e5d-a512071934d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:14.354338 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.354201 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5aba07a-x585x\" (UID: \"6d36cc25-4820-4388-8e5d-a512071934d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:14.354788 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.354769 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5aba07a-x585x\" (UID: \"6d36cc25-4820-4388-8e5d-a512071934d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:14.356844 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.356804 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5aba07a-x585x\" (UID: \"6d36cc25-4820-4388-8e5d-a512071934d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:14.368396 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.368374 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grbxx\" (UniqueName: \"kubernetes.io/projected/6d36cc25-4820-4388-8e5d-a512071934d3-kube-api-access-grbxx\") pod \"maas-api-revoke-sub-d5aba07a-x585x\" (UID: \"6d36cc25-4820-4388-8e5d-a512071934d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:14.458700 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.458632 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:14.741399 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:14.741372 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x"] Oct 31 00:02:14.743793 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:02:14.743768 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d36cc25_4820_4388_8e5d_a512071934d3.slice/crio-b42b352c4f71e0bbb9a6b58b999d1d22d79c85719d1d5976a827865c99003908 WatchSource:0}: Error finding container b42b352c4f71e0bbb9a6b58b999d1d22d79c85719d1d5976a827865c99003908: Status 404 returned error can't find the container with id b42b352c4f71e0bbb9a6b58b999d1d22d79c85719d1d5976a827865c99003908 Oct 31 00:02:15.238923 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:15.238890 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" event={"ID":"6d36cc25-4820-4388-8e5d-a512071934d3","Type":"ContainerStarted","Data":"d5a827ee07aaeaae63dcc7224ff9061110146fb94db7c3ae5515a95afb9c488d"} Oct 31 00:02:15.239324 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:15.238945 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" event={"ID":"6d36cc25-4820-4388-8e5d-a512071934d3","Type":"ContainerStarted","Data":"b42b352c4f71e0bbb9a6b58b999d1d22d79c85719d1d5976a827865c99003908"} Oct 31 00:02:15.258780 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:15.258696 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" podStartSLOduration=1.258683309 podStartE2EDuration="1.258683309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:15.257831132 +0000 UTC m=+2546.776832236" watchObservedRunningTime="2026-10-31 00:02:15.258683309 +0000 UTC m=+2546.777684398" Oct 31 00:02:16.246971 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:16.246932 7570 generic.go:356] "Generic (PLEG): container finished" podID="6d36cc25-4820-4388-8e5d-a512071934d3" containerID="d5a827ee07aaeaae63dcc7224ff9061110146fb94db7c3ae5515a95afb9c488d" exitCode=0 Oct 31 00:02:16.246971 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:16.246968 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" event={"ID":"6d36cc25-4820-4388-8e5d-a512071934d3","Type":"ContainerDied","Data":"d5a827ee07aaeaae63dcc7224ff9061110146fb94db7c3ae5515a95afb9c488d"} Oct 31 00:02:17.387434 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:17.387408 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:17.497700 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:17.497662 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d36cc25-4820-4388-8e5d-a512071934d3-kube-api-access-grbxx\" (UniqueName: \"kubernetes.io/projected/6d36cc25-4820-4388-8e5d-a512071934d3-kube-api-access-grbxx\") pod \"6d36cc25-4820-4388-8e5d-a512071934d3\" (UID: \"6d36cc25-4820-4388-8e5d-a512071934d3\") " Oct 31 00:02:17.497867 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:17.497789 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-openshift-ca\") pod \"6d36cc25-4820-4388-8e5d-a512071934d3\" (UID: \"6d36cc25-4820-4388-8e5d-a512071934d3\") " Oct 31 00:02:17.497946 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:17.497885 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-xks-ca\") pod \"6d36cc25-4820-4388-8e5d-a512071934d3\" (UID: \"6d36cc25-4820-4388-8e5d-a512071934d3\") " Oct 31 00:02:17.498210 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:17.498181 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-openshift-ca" pod "6d36cc25-4820-4388-8e5d-a512071934d3" (UID: "6d36cc25-4820-4388-8e5d-a512071934d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:17.498346 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:17.498326 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:17.500186 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:17.500157 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-xks-ca" pod "6d36cc25-4820-4388-8e5d-a512071934d3" (UID: "6d36cc25-4820-4388-8e5d-a512071934d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:17.500278 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:17.500161 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d36cc25-4820-4388-8e5d-a512071934d3-kube-api-access-grbxx" pod "6d36cc25-4820-4388-8e5d-a512071934d3" (UID: "6d36cc25-4820-4388-8e5d-a512071934d3"). InnerVolumeSpecName "kube-api-access-grbxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:17.599729 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:17.599689 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grbxx\" (UniqueName: \"kubernetes.io/projected/6d36cc25-4820-4388-8e5d-a512071934d3-kube-api-access-grbxx\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:17.599729 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:17.599723 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d36cc25-4820-4388-8e5d-a512071934d3-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:18.261355 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:18.261321 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" Oct 31 00:02:18.261355 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:18.261337 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x" event={"ID":"6d36cc25-4820-4388-8e5d-a512071934d3","Type":"ContainerDied","Data":"b42b352c4f71e0bbb9a6b58b999d1d22d79c85719d1d5976a827865c99003908"} Oct 31 00:02:18.261555 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:18.261388 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b42b352c4f71e0bbb9a6b58b999d1d22d79c85719d1d5976a827865c99003908" Oct 31 00:02:24.945346 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:24.945311 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26"] Oct 31 00:02:24.945819 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:24.945796 7570 cpu_manager.go:414] "containerMap: removing container" podUID="6d36cc25-4820-4388-8e5d-a512071934d3" containerName="revoke-keys" Oct 31 00:02:24.945819 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:24.945808 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d36cc25-4820-4388-8e5d-a512071934d3" containerName="revoke-keys" Oct 31 00:02:24.945906 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:24.945896 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d36cc25-4820-4388-8e5d-a512071934d3" containerName="revoke-keys" Oct 31 00:02:24.949499 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:24.949480 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:24.957588 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:24.957565 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:02:24.958664 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:24.958639 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:02:24.958664 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:24.958657 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26"] Oct 31 00:02:25.021109 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.021079 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:02:25.021109 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.021103 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:02:25.078977 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.078951 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26\" (UID: \"80ed1470-002a-4704-a7c8-2c0692bf17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:25.079108 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.079025 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2hg\" (UniqueName: \"kubernetes.io/projected/80ed1470-002a-4704-a7c8-2c0692bf17fc-kube-api-access-bf2hg\") pod \"maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26\" (UID: \"80ed1470-002a-4704-a7c8-2c0692bf17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:25.079181 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.079100 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26\" (UID: \"80ed1470-002a-4704-a7c8-2c0692bf17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:25.179681 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.179656 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2hg\" (UniqueName: \"kubernetes.io/projected/80ed1470-002a-4704-a7c8-2c0692bf17fc-kube-api-access-bf2hg\") pod \"maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26\" (UID: \"80ed1470-002a-4704-a7c8-2c0692bf17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:25.179795 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.179683 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26\" (UID: \"80ed1470-002a-4704-a7c8-2c0692bf17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:25.179795 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.179753 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26\" (UID: \"80ed1470-002a-4704-a7c8-2c0692bf17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:25.180339 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.180320 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26\" (UID: \"80ed1470-002a-4704-a7c8-2c0692bf17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:25.182205 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.182187 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26\" (UID: \"80ed1470-002a-4704-a7c8-2c0692bf17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:25.195577 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.195533 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2hg\" (UniqueName: \"kubernetes.io/projected/80ed1470-002a-4704-a7c8-2c0692bf17fc-kube-api-access-bf2hg\") pod \"maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26\" (UID: \"80ed1470-002a-4704-a7c8-2c0692bf17fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:25.267389 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.267366 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:25.550696 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:25.550669 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26"] Oct 31 00:02:25.551869 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:02:25.551849 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ed1470_002a_4704_a7c8_2c0692bf17fc.slice/crio-fbf05577d20fc44d21a0d144fd53c57295608b3a18b48707d75e98df145ebff4 WatchSource:0}: Error finding container fbf05577d20fc44d21a0d144fd53c57295608b3a18b48707d75e98df145ebff4: Status 404 returned error can't find the container with id fbf05577d20fc44d21a0d144fd53c57295608b3a18b48707d75e98df145ebff4 Oct 31 00:02:26.311685 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:26.311638 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" event={"ID":"80ed1470-002a-4704-a7c8-2c0692bf17fc","Type":"ContainerStarted","Data":"e0a2a5647b3290be3272d54f372d86b760ab4b85af94f2e3f1673f340b46f4c4"} Oct 31 00:02:26.312177 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:26.311710 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" event={"ID":"80ed1470-002a-4704-a7c8-2c0692bf17fc","Type":"ContainerStarted","Data":"fbf05577d20fc44d21a0d144fd53c57295608b3a18b48707d75e98df145ebff4"} Oct 31 00:02:26.332234 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:26.332176 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" podStartSLOduration=2.332163644 podStartE2EDuration="2.332163644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:26.3294103 +0000 UTC m=+2557.848411416" watchObservedRunningTime="2026-10-31 00:02:26.332163644 +0000 UTC m=+2557.851164734" Oct 31 00:02:27.320422 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:27.320383 7570 generic.go:356] "Generic (PLEG): container finished" podID="80ed1470-002a-4704-a7c8-2c0692bf17fc" containerID="e0a2a5647b3290be3272d54f372d86b760ab4b85af94f2e3f1673f340b46f4c4" exitCode=0 Oct 31 00:02:27.320908 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:27.320504 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" event={"ID":"80ed1470-002a-4704-a7c8-2c0692bf17fc","Type":"ContainerDied","Data":"e0a2a5647b3290be3272d54f372d86b760ab4b85af94f2e3f1673f340b46f4c4"} Oct 31 00:02:28.464942 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:28.464919 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:28.613070 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:28.613046 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-xks-ca\") pod \"80ed1470-002a-4704-a7c8-2c0692bf17fc\" (UID: \"80ed1470-002a-4704-a7c8-2c0692bf17fc\") " Oct 31 00:02:28.613210 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:28.613170 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-openshift-ca\") pod \"80ed1470-002a-4704-a7c8-2c0692bf17fc\" (UID: \"80ed1470-002a-4704-a7c8-2c0692bf17fc\") " Oct 31 00:02:28.613210 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:28.613200 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ed1470-002a-4704-a7c8-2c0692bf17fc-kube-api-access-bf2hg\" (UniqueName: \"kubernetes.io/projected/80ed1470-002a-4704-a7c8-2c0692bf17fc-kube-api-access-bf2hg\") pod \"80ed1470-002a-4704-a7c8-2c0692bf17fc\" (UID: \"80ed1470-002a-4704-a7c8-2c0692bf17fc\") " Oct 31 00:02:28.613531 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:28.613511 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-openshift-ca" pod "80ed1470-002a-4704-a7c8-2c0692bf17fc" (UID: "80ed1470-002a-4704-a7c8-2c0692bf17fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:28.613723 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:28.613668 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:28.615349 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:28.615318 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-xks-ca" pod "80ed1470-002a-4704-a7c8-2c0692bf17fc" (UID: "80ed1470-002a-4704-a7c8-2c0692bf17fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:28.615453 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:28.615403 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ed1470-002a-4704-a7c8-2c0692bf17fc-kube-api-access-bf2hg" pod "80ed1470-002a-4704-a7c8-2c0692bf17fc" (UID: "80ed1470-002a-4704-a7c8-2c0692bf17fc"). InnerVolumeSpecName "kube-api-access-bf2hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:28.715103 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:28.715073 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80ed1470-002a-4704-a7c8-2c0692bf17fc-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:28.715103 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:28.715101 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf2hg\" (UniqueName: \"kubernetes.io/projected/80ed1470-002a-4704-a7c8-2c0692bf17fc-kube-api-access-bf2hg\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:29.334750 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:29.334726 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" Oct 31 00:02:29.334926 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:29.334724 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26" event={"ID":"80ed1470-002a-4704-a7c8-2c0692bf17fc","Type":"ContainerDied","Data":"fbf05577d20fc44d21a0d144fd53c57295608b3a18b48707d75e98df145ebff4"} Oct 31 00:02:29.334926 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:29.334870 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbf05577d20fc44d21a0d144fd53c57295608b3a18b48707d75e98df145ebff4" Oct 31 00:02:29.411400 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:02:29.411366 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:02:36.981244 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:36.981208 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26"] Oct 31 00:02:36.983952 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:36.983934 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b38a4-xsb26"] Oct 31 00:02:38.602290 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:38.602260 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ed1470-002a-4704-a7c8-2c0692bf17fc" path="/var/lib/kubelet/pods/80ed1470-002a-4704-a7c8-2c0692bf17fc/volumes" Oct 31 00:02:43.383186 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:43.383157 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log" Oct 31 00:02:45.435766 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.435730 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb"] Oct 31 00:02:45.438142 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.436186 7570 cpu_manager.go:414] "containerMap: removing container" podUID="80ed1470-002a-4704-a7c8-2c0692bf17fc" containerName="revoke-keys" Oct 31 00:02:45.438142 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.436197 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ed1470-002a-4704-a7c8-2c0692bf17fc" containerName="revoke-keys" Oct 31 00:02:45.438142 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.436279 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ed1470-002a-4704-a7c8-2c0692bf17fc" containerName="revoke-keys" Oct 31 00:02:45.439446 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.439430 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:45.446252 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.446228 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:02:45.446905 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.446878 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:02:45.449849 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.449829 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb"] Oct 31 00:02:45.474205 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.474184 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:02:45.474327 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.474314 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:02:45.487128 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.487105 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb\" (UID: \"3d6d17d3-fc5e-4155-8021-57be8a668a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:45.487253 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.487133 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prj24\" (UniqueName: \"kubernetes.io/projected/3d6d17d3-fc5e-4155-8021-57be8a668a58-kube-api-access-prj24\") pod \"maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb\" (UID: \"3d6d17d3-fc5e-4155-8021-57be8a668a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:45.487336 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.487317 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb\" (UID: \"3d6d17d3-fc5e-4155-8021-57be8a668a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:45.524108 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.524090 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log" Oct 31 00:02:45.587957 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.587933 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb\" (UID: \"3d6d17d3-fc5e-4155-8021-57be8a668a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:45.588049 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.587986 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb\" (UID: \"3d6d17d3-fc5e-4155-8021-57be8a668a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:45.588110 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.588092 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prj24\" (UniqueName: \"kubernetes.io/projected/3d6d17d3-fc5e-4155-8021-57be8a668a58-kube-api-access-prj24\") pod \"maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb\" (UID: \"3d6d17d3-fc5e-4155-8021-57be8a668a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:45.588559 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.588542 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb\" (UID: \"3d6d17d3-fc5e-4155-8021-57be8a668a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:45.590559 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.590544 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb\" (UID: \"3d6d17d3-fc5e-4155-8021-57be8a668a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:45.613247 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.613205 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prj24\" (UniqueName: \"kubernetes.io/projected/3d6d17d3-fc5e-4155-8021-57be8a668a58-kube-api-access-prj24\") pod \"maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb\" (UID: \"3d6d17d3-fc5e-4155-8021-57be8a668a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:45.755231 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:45.755157 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:46.036055 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:46.036028 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb"] Oct 31 00:02:46.037846 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:02:46.037815 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6d17d3_fc5e_4155_8021_57be8a668a58.slice/crio-4c3878e234faf8248eaed8b1db606443550c4fc1b42aa657c778dfea2270f3c2 WatchSource:0}: Error finding container 4c3878e234faf8248eaed8b1db606443550c4fc1b42aa657c778dfea2270f3c2: Status 404 returned error can't find the container with id 4c3878e234faf8248eaed8b1db606443550c4fc1b42aa657c778dfea2270f3c2 Oct 31 00:02:46.461678 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:46.461646 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" event={"ID":"3d6d17d3-fc5e-4155-8021-57be8a668a58","Type":"ContainerStarted","Data":"3997980767aebc5329d79db450bf6df6bfb0528eec37ab42b3f3163ffa96a879"} Oct 31 00:02:46.462248 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:46.461700 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" event={"ID":"3d6d17d3-fc5e-4155-8021-57be8a668a58","Type":"ContainerStarted","Data":"4c3878e234faf8248eaed8b1db606443550c4fc1b42aa657c778dfea2270f3c2"} Oct 31 00:02:46.481332 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:46.481265 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" podStartSLOduration=1.481202633 podStartE2EDuration="1.481202633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:46.480891705 +0000 UTC m=+2577.999892795" watchObservedRunningTime="2026-10-31 00:02:46.481202633 +0000 UTC m=+2578.000203723" Oct 31 00:02:47.470279 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:47.470243 7570 generic.go:356] "Generic (PLEG): container finished" podID="3d6d17d3-fc5e-4155-8021-57be8a668a58" containerID="3997980767aebc5329d79db450bf6df6bfb0528eec37ab42b3f3163ffa96a879" exitCode=0 Oct 31 00:02:47.470725 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:47.470332 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" event={"ID":"3d6d17d3-fc5e-4155-8021-57be8a668a58","Type":"ContainerDied","Data":"3997980767aebc5329d79db450bf6df6bfb0528eec37ab42b3f3163ffa96a879"} Oct 31 00:02:48.623909 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:48.623880 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:48.720922 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:48.720892 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6d17d3-fc5e-4155-8021-57be8a668a58-kube-api-access-prj24\" (UniqueName: \"kubernetes.io/projected/3d6d17d3-fc5e-4155-8021-57be8a668a58-kube-api-access-prj24\") pod \"3d6d17d3-fc5e-4155-8021-57be8a668a58\" (UID: \"3d6d17d3-fc5e-4155-8021-57be8a668a58\") " Oct 31 00:02:48.721084 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:48.720950 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-openshift-ca\") pod \"3d6d17d3-fc5e-4155-8021-57be8a668a58\" (UID: \"3d6d17d3-fc5e-4155-8021-57be8a668a58\") " Oct 31 00:02:48.721084 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:48.721026 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-xks-ca\") pod \"3d6d17d3-fc5e-4155-8021-57be8a668a58\" (UID: \"3d6d17d3-fc5e-4155-8021-57be8a668a58\") " Oct 31 00:02:48.721353 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:48.721319 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-openshift-ca" pod "3d6d17d3-fc5e-4155-8021-57be8a668a58" (UID: "3d6d17d3-fc5e-4155-8021-57be8a668a58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:02:48.721580 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:48.721560 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:48.723178 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:48.723155 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-xks-ca" pod "3d6d17d3-fc5e-4155-8021-57be8a668a58" (UID: "3d6d17d3-fc5e-4155-8021-57be8a668a58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:02:48.723481 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:48.723450 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6d17d3-fc5e-4155-8021-57be8a668a58-kube-api-access-prj24" pod "3d6d17d3-fc5e-4155-8021-57be8a668a58" (UID: "3d6d17d3-fc5e-4155-8021-57be8a668a58"). InnerVolumeSpecName "kube-api-access-prj24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:48.822307 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:48.822280 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prj24\" (UniqueName: \"kubernetes.io/projected/3d6d17d3-fc5e-4155-8021-57be8a668a58-kube-api-access-prj24\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:48.822307 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:48.822307 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6d17d3-fc5e-4155-8021-57be8a668a58-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:49.487690 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:49.487648 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" event={"ID":"3d6d17d3-fc5e-4155-8021-57be8a668a58","Type":"ContainerDied","Data":"4c3878e234faf8248eaed8b1db606443550c4fc1b42aa657c778dfea2270f3c2"} Oct 31 00:02:49.487863 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:49.487701 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c3878e234faf8248eaed8b1db606443550c4fc1b42aa657c778dfea2270f3c2" Oct 31 00:02:49.487863 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:49.487754 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb" Oct 31 00:02:57.114607 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:57.114575 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb"] Oct 31 00:02:57.118361 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:57.118340 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e5934f-nk5lb"] Oct 31 00:02:58.600573 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:02:58.600538 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d6d17d3-fc5e-4155-8021-57be8a668a58" path="/var/lib/kubelet/pods/3d6d17d3-fc5e-4155-8021-57be8a668a58/volumes" Oct 31 00:02:59.428796 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:02:59.428742 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:03:06.019086 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.019051 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j"] Oct 31 00:03:06.019551 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.019537 7570 cpu_manager.go:414] "containerMap: removing container" podUID="3d6d17d3-fc5e-4155-8021-57be8a668a58" containerName="revoke-keys" Oct 31 00:03:06.019604 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.019552 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6d17d3-fc5e-4155-8021-57be8a668a58" containerName="revoke-keys" Oct 31 00:03:06.019639 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.019617 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6d17d3-fc5e-4155-8021-57be8a668a58" containerName="revoke-keys" Oct 31 00:03:06.024036 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.024020 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:06.031975 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.031947 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j"] Oct 31 00:03:06.065571 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.065543 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:03:06.065706 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.065670 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:03:06.090866 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.090839 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:03:06.091044 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.091028 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:03:06.103132 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.103112 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j\" (UID: \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:06.103229 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.103168 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsdj9\" (UniqueName: \"kubernetes.io/projected/599a3946-97e4-439b-b2e7-1f1ce2eb6125-kube-api-access-fsdj9\") pod \"maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j\" (UID: \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:06.103329 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.103311 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j\" (UID: \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:06.204451 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.204415 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j\" (UID: \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:06.204653 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.204514 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsdj9\" (UniqueName: \"kubernetes.io/projected/599a3946-97e4-439b-b2e7-1f1ce2eb6125-kube-api-access-fsdj9\") pod \"maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j\" (UID: \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:06.204653 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.204562 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j\" (UID: \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:06.205195 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.205170 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j\" (UID: \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:06.207009 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.206987 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j\" (UID: \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:06.216744 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.216724 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsdj9\" (UniqueName: \"kubernetes.io/projected/599a3946-97e4-439b-b2e7-1f1ce2eb6125-kube-api-access-fsdj9\") pod \"maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j\" (UID: \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:06.339953 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.339924 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:06.625217 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:06.625185 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j"] Oct 31 00:03:06.626270 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:03:06.626241 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599a3946_97e4_439b_b2e7_1f1ce2eb6125.slice/crio-de593e0fe8c6ebe80f6a1d14c062907ce99fab2ec0303d6a8af5198c20b5161c WatchSource:0}: Error finding container de593e0fe8c6ebe80f6a1d14c062907ce99fab2ec0303d6a8af5198c20b5161c: Status 404 returned error can't find the container with id de593e0fe8c6ebe80f6a1d14c062907ce99fab2ec0303d6a8af5198c20b5161c Oct 31 00:03:07.614047 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:07.614022 7570 generic.go:356] "Generic (PLEG): container finished" podID="599a3946-97e4-439b-b2e7-1f1ce2eb6125" containerID="116039e34d9c351dc9ec13055520dda8c46a27a7624044cc42d6338aaee55978" exitCode=0 Oct 31 00:03:07.614390 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:07.614116 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" event={"ID":"599a3946-97e4-439b-b2e7-1f1ce2eb6125","Type":"ContainerDied","Data":"116039e34d9c351dc9ec13055520dda8c46a27a7624044cc42d6338aaee55978"} Oct 31 00:03:07.614390 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:07.614165 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" event={"ID":"599a3946-97e4-439b-b2e7-1f1ce2eb6125","Type":"ContainerStarted","Data":"de593e0fe8c6ebe80f6a1d14c062907ce99fab2ec0303d6a8af5198c20b5161c"} Oct 31 00:03:08.759929 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:08.759902 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:08.931266 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:08.931189 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-xks-ca\") pod \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\" (UID: \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\") " Oct 31 00:03:08.931423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:08.931298 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-openshift-ca\") pod \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\" (UID: \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\") " Oct 31 00:03:08.931423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:08.931378 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599a3946-97e4-439b-b2e7-1f1ce2eb6125-kube-api-access-fsdj9\" (UniqueName: \"kubernetes.io/projected/599a3946-97e4-439b-b2e7-1f1ce2eb6125-kube-api-access-fsdj9\") pod \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\" (UID: \"599a3946-97e4-439b-b2e7-1f1ce2eb6125\") " Oct 31 00:03:08.931679 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:08.931651 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-openshift-ca" pod "599a3946-97e4-439b-b2e7-1f1ce2eb6125" (UID: "599a3946-97e4-439b-b2e7-1f1ce2eb6125"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:08.933430 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:08.933399 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-xks-ca" pod "599a3946-97e4-439b-b2e7-1f1ce2eb6125" (UID: "599a3946-97e4-439b-b2e7-1f1ce2eb6125"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:08.933571 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:08.933481 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599a3946-97e4-439b-b2e7-1f1ce2eb6125-kube-api-access-fsdj9" pod "599a3946-97e4-439b-b2e7-1f1ce2eb6125" (UID: "599a3946-97e4-439b-b2e7-1f1ce2eb6125"). InnerVolumeSpecName "kube-api-access-fsdj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:09.032376 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:09.032350 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:09.032376 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:09.032377 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a3946-97e4-439b-b2e7-1f1ce2eb6125-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:09.032524 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:09.032387 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsdj9\" (UniqueName: \"kubernetes.io/projected/599a3946-97e4-439b-b2e7-1f1ce2eb6125-kube-api-access-fsdj9\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:09.632115 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:09.632086 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" Oct 31 00:03:09.632275 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:09.632088 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j" event={"ID":"599a3946-97e4-439b-b2e7-1f1ce2eb6125","Type":"ContainerDied","Data":"de593e0fe8c6ebe80f6a1d14c062907ce99fab2ec0303d6a8af5198c20b5161c"} Oct 31 00:03:09.632275 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:09.632213 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de593e0fe8c6ebe80f6a1d14c062907ce99fab2ec0303d6a8af5198c20b5161c" Oct 31 00:03:10.726494 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:10.726420 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r"] Oct 31 00:03:16.553271 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.553230 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv"] Oct 31 00:03:16.553934 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.553911 7570 cpu_manager.go:414] "containerMap: removing container" podUID="599a3946-97e4-439b-b2e7-1f1ce2eb6125" containerName="revoke-keys" Oct 31 00:03:16.554036 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.553938 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599a3946-97e4-439b-b2e7-1f1ce2eb6125" containerName="revoke-keys" Oct 31 00:03:16.554126 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.554113 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="599a3946-97e4-439b-b2e7-1f1ce2eb6125" containerName="revoke-keys" Oct 31 00:03:16.557546 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.557525 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:16.563163 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.563143 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:03:16.563292 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.563249 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:03:16.563354 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.563313 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:03:16.563703 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.563674 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:03:16.566091 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.566050 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv"] Oct 31 00:03:16.712731 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.712703 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fxzj\" (UniqueName: \"kubernetes.io/projected/c53048d8-1746-42ae-91a4-f160a2948c62-kube-api-access-6fxzj\") pod \"maas-api-revoke-sub-5510eb64-dmzxv\" (UID: \"c53048d8-1746-42ae-91a4-f160a2948c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:16.712904 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.712764 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5510eb64-dmzxv\" (UID: \"c53048d8-1746-42ae-91a4-f160a2948c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:16.712982 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.712900 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5510eb64-dmzxv\" (UID: \"c53048d8-1746-42ae-91a4-f160a2948c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:16.813938 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.813874 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5510eb64-dmzxv\" (UID: \"c53048d8-1746-42ae-91a4-f160a2948c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:16.814045 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.813940 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5510eb64-dmzxv\" (UID: \"c53048d8-1746-42ae-91a4-f160a2948c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:16.814045 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.814000 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fxzj\" (UniqueName: \"kubernetes.io/projected/c53048d8-1746-42ae-91a4-f160a2948c62-kube-api-access-6fxzj\") pod \"maas-api-revoke-sub-5510eb64-dmzxv\" (UID: \"c53048d8-1746-42ae-91a4-f160a2948c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:16.814489 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.814443 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5510eb64-dmzxv\" (UID: \"c53048d8-1746-42ae-91a4-f160a2948c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:16.816584 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.816557 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5510eb64-dmzxv\" (UID: \"c53048d8-1746-42ae-91a4-f160a2948c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:16.826332 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.826310 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fxzj\" (UniqueName: \"kubernetes.io/projected/c53048d8-1746-42ae-91a4-f160a2948c62-kube-api-access-6fxzj\") pod \"maas-api-revoke-sub-5510eb64-dmzxv\" (UID: \"c53048d8-1746-42ae-91a4-f160a2948c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:16.873221 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:16.873201 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:17.152076 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:17.152045 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv"] Oct 31 00:03:17.152675 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:03:17.152645 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53048d8_1746_42ae_91a4_f160a2948c62.slice/crio-6e5d3ed6996b70fa4ff8bd0e239c21488e3e83020d6954b9f764b66ff1185327 WatchSource:0}: Error finding container 6e5d3ed6996b70fa4ff8bd0e239c21488e3e83020d6954b9f764b66ff1185327: Status 404 returned error can't find the container with id 6e5d3ed6996b70fa4ff8bd0e239c21488e3e83020d6954b9f764b66ff1185327 Oct 31 00:03:17.682053 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:17.682013 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" event={"ID":"c53048d8-1746-42ae-91a4-f160a2948c62","Type":"ContainerStarted","Data":"e6d491fa38ada313319950abb7dee727eb256cbbb499627f05a6b20d5ad59f01"} Oct 31 00:03:17.682485 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:17.682081 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" event={"ID":"c53048d8-1746-42ae-91a4-f160a2948c62","Type":"ContainerStarted","Data":"6e5d3ed6996b70fa4ff8bd0e239c21488e3e83020d6954b9f764b66ff1185327"} Oct 31 00:03:17.702339 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:17.702280 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" podStartSLOduration=1.702267537 podStartE2EDuration="1.702267537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:17.700654463 +0000 UTC m=+2609.219655554" watchObservedRunningTime="2026-10-31 00:03:17.702267537 +0000 UTC m=+2609.221268626" Oct 31 00:03:18.691885 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:18.691845 7570 generic.go:356] "Generic (PLEG): container finished" podID="c53048d8-1746-42ae-91a4-f160a2948c62" containerID="e6d491fa38ada313319950abb7dee727eb256cbbb499627f05a6b20d5ad59f01" exitCode=0 Oct 31 00:03:18.692283 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:18.691939 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" event={"ID":"c53048d8-1746-42ae-91a4-f160a2948c62","Type":"ContainerDied","Data":"e6d491fa38ada313319950abb7dee727eb256cbbb499627f05a6b20d5ad59f01"} Oct 31 00:03:19.862676 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:19.862651 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:19.947052 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:19.947016 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-openshift-ca\") pod \"c53048d8-1746-42ae-91a4-f160a2948c62\" (UID: \"c53048d8-1746-42ae-91a4-f160a2948c62\") " Oct 31 00:03:19.947210 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:19.947126 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-xks-ca\") pod \"c53048d8-1746-42ae-91a4-f160a2948c62\" (UID: \"c53048d8-1746-42ae-91a4-f160a2948c62\") " Oct 31 00:03:19.947210 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:19.947172 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53048d8-1746-42ae-91a4-f160a2948c62-kube-api-access-6fxzj\" (UniqueName: \"kubernetes.io/projected/c53048d8-1746-42ae-91a4-f160a2948c62-kube-api-access-6fxzj\") pod \"c53048d8-1746-42ae-91a4-f160a2948c62\" (UID: \"c53048d8-1746-42ae-91a4-f160a2948c62\") " Oct 31 00:03:19.947449 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:19.947422 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-openshift-ca" pod "c53048d8-1746-42ae-91a4-f160a2948c62" (UID: "c53048d8-1746-42ae-91a4-f160a2948c62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:19.947701 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:19.947679 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:19.949679 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:19.949608 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-xks-ca" pod "c53048d8-1746-42ae-91a4-f160a2948c62" (UID: "c53048d8-1746-42ae-91a4-f160a2948c62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:19.949679 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:19.949639 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53048d8-1746-42ae-91a4-f160a2948c62-kube-api-access-6fxzj" pod "c53048d8-1746-42ae-91a4-f160a2948c62" (UID: "c53048d8-1746-42ae-91a4-f160a2948c62"). InnerVolumeSpecName "kube-api-access-6fxzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:20.048961 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:20.048925 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53048d8-1746-42ae-91a4-f160a2948c62-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:20.048961 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:20.048960 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fxzj\" (UniqueName: \"kubernetes.io/projected/c53048d8-1746-42ae-91a4-f160a2948c62-kube-api-access-6fxzj\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:20.705754 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:20.705723 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" Oct 31 00:03:20.705913 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:20.705749 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv" event={"ID":"c53048d8-1746-42ae-91a4-f160a2948c62","Type":"ContainerDied","Data":"6e5d3ed6996b70fa4ff8bd0e239c21488e3e83020d6954b9f764b66ff1185327"} Oct 31 00:03:20.705913 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:20.705799 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e5d3ed6996b70fa4ff8bd0e239c21488e3e83020d6954b9f764b66ff1185327" Oct 31 00:03:25.749535 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:25.749495 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" containerID="cri-o://1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca" gracePeriod=45 Oct 31 00:03:25.755105 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:25.755073 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": dial tcp 10.134.0.76:8000: connect: connection refused" Oct 31 00:03:25.910331 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:25.910295 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g"] Oct 31 00:03:25.910635 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:25.910577 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" containerID="cri-o://9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855" gracePeriod=30 Oct 31 00:03:25.932240 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:25.932194 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7"] Oct 31 00:03:25.932494 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:25.932444 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" containerID="cri-o://4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008" gracePeriod=30 Oct 31 00:03:26.045323 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.045301 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 31 00:03:26.112308 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.112262 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-tmp-dir\") pod \"809bef14-259b-4a21-b170-a6a701e00e6c\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " Oct 31 00:03:26.112515 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.112360 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-model-cache\") pod \"809bef14-259b-4a21-b170-a6a701e00e6c\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " Oct 31 00:03:26.112687 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.112662 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-tmp-dir" pod "809bef14-259b-4a21-b170-a6a701e00e6c" (UID: "809bef14-259b-4a21-b170-a6a701e00e6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:26.112743 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.112683 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-model-cache" pod "809bef14-259b-4a21-b170-a6a701e00e6c" (UID: "809bef14-259b-4a21-b170-a6a701e00e6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:26.112930 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.112908 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-home\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-home\") pod \"809bef14-259b-4a21-b170-a6a701e00e6c\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " Oct 31 00:03:26.112975 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.112964 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-dshm\") pod \"809bef14-259b-4a21-b170-a6a701e00e6c\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " Oct 31 00:03:26.113015 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.113002 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809bef14-259b-4a21-b170-a6a701e00e6c-kube-api-access-dfwxh\" (UniqueName: \"kubernetes.io/projected/809bef14-259b-4a21-b170-a6a701e00e6c-kube-api-access-dfwxh\") pod \"809bef14-259b-4a21-b170-a6a701e00e6c\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " Oct 31 00:03:26.113057 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.113040 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/809bef14-259b-4a21-b170-a6a701e00e6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/809bef14-259b-4a21-b170-a6a701e00e6c-tls-certs\") pod \"809bef14-259b-4a21-b170-a6a701e00e6c\" (UID: \"809bef14-259b-4a21-b170-a6a701e00e6c\") " Oct 31 00:03:26.113205 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.113178 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-home" pod "809bef14-259b-4a21-b170-a6a701e00e6c" (UID: "809bef14-259b-4a21-b170-a6a701e00e6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:26.113532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.113510 7570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-tmp-dir\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:26.113532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.113534 7570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-model-cache\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:26.113725 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.113547 7570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-home\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:26.116792 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.116759 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-dshm" pod "809bef14-259b-4a21-b170-a6a701e00e6c" (UID: "809bef14-259b-4a21-b170-a6a701e00e6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:26.117303 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.117284 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/809bef14-259b-4a21-b170-a6a701e00e6c-tls-certs" pod "809bef14-259b-4a21-b170-a6a701e00e6c" (UID: "809bef14-259b-4a21-b170-a6a701e00e6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:26.125585 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.125561 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809bef14-259b-4a21-b170-a6a701e00e6c-kube-api-access-dfwxh" pod "809bef14-259b-4a21-b170-a6a701e00e6c" (UID: "809bef14-259b-4a21-b170-a6a701e00e6c"). InnerVolumeSpecName "kube-api-access-dfwxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:26.215025 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.214992 7570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/809bef14-259b-4a21-b170-a6a701e00e6c-dshm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:26.215025 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.215024 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfwxh\" (UniqueName: \"kubernetes.io/projected/809bef14-259b-4a21-b170-a6a701e00e6c-kube-api-access-dfwxh\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:26.215025 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.215033 7570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809bef14-259b-4a21-b170-a6a701e00e6c-tls-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:26.750447 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.750418 7570 generic.go:356] "Generic (PLEG): container finished" podID="809bef14-259b-4a21-b170-a6a701e00e6c" containerID="1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca" exitCode=0 Oct 31 00:03:26.750856 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.750485 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" Oct 31 00:03:26.750856 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.750505 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" event={"ID":"809bef14-259b-4a21-b170-a6a701e00e6c","Type":"ContainerDied","Data":"1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca"} Oct 31 00:03:26.750856 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.750546 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r" event={"ID":"809bef14-259b-4a21-b170-a6a701e00e6c","Type":"ContainerDied","Data":"178cf9eaeaccbae96aa052845e91cddb36b2fe3882ff181d6bfff122449379d3"} Oct 31 00:03:26.750856 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.750565 7570 scope.go:122] "RemoveContainer" containerID="1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca" Oct 31 00:03:26.761764 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.761745 7570 scope.go:122] "RemoveContainer" containerID="1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca" Oct 31 00:03:26.762021 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:03:26.762000 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca\": container with ID starting with 1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca not found: ID does not exist" containerID="1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca" Oct 31 00:03:26.762062 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.762029 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca"} err="failed to get container status \"1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca\": rpc error: code = NotFound desc = could not find container \"1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca\": container with ID starting with 1a69381cff9c4f1425a62b703ee29e750d8b4e4d194129548a7595d10f8d12ca not found: ID does not exist" Oct 31 00:03:26.772973 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.772944 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r"] Oct 31 00:03:26.781118 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:26.781099 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-dd3a15/distinct-2-w5-dd3a15-kserve-86bb9549d9-m7g9r"] Oct 31 00:03:28.142970 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:28.142936 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-t7b8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:03:28.143445 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:28.142991 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:03:28.155673 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:28.155644 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:03:28.155790 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:28.155698 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:03:28.601035 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:28.601005 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" path="/var/lib/kubelet/pods/809bef14-259b-4a21-b170-a6a701e00e6c/volumes" Oct 31 00:03:29.412344 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:03:29.412304 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:03:31.185434 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.185413 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 31 00:03:31.223800 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.223773 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 31 00:03:31.264717 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.264691 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-kube-api-access-2zqds\" (UniqueName: \"kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-kube-api-access-2zqds\") pod \"02d621d9-9a83-45c7-bef8-9255f543c6f0\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " Oct 31 00:03:31.264858 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.264741 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-socket\") pod \"02d621d9-9a83-45c7-bef8-9255f543c6f0\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " Oct 31 00:03:31.264858 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.264809 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31585445-5877-4947-8df5-f30317948127-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31585445-5877-4947-8df5-f30317948127-istio-podinfo\") pod \"31585445-5877-4947-8df5-f30317948127\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " Oct 31 00:03:31.264977 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.264856 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-socket\") pod \"31585445-5877-4947-8df5-f30317948127\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " Oct 31 00:03:31.264977 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.264895 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-istio-token\" (UniqueName: \"kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-istio-token\") pod \"31585445-5877-4947-8df5-f30317948127\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " Oct 31 00:03:31.264977 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.264928 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-certs\") pod \"31585445-5877-4947-8df5-f30317948127\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " Oct 31 00:03:31.264977 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.264956 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-token\" (UniqueName: \"kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-token\") pod \"02d621d9-9a83-45c7-bef8-9255f543c6f0\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " Oct 31 00:03:31.265180 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265001 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-credential-socket\") pod \"31585445-5877-4947-8df5-f30317948127\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " Oct 31 00:03:31.265180 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265029 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-envoy\") pod \"02d621d9-9a83-45c7-bef8-9255f543c6f0\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " Oct 31 00:03:31.265180 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265059 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-socket" pod "02d621d9-9a83-45c7-bef8-9255f543c6f0" (UID: "02d621d9-9a83-45c7-bef8-9255f543c6f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:31.265180 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265081 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-kube-api-access-fssz2\" (UniqueName: \"kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-kube-api-access-fssz2\") pod \"31585445-5877-4947-8df5-f30317948127\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " Oct 31 00:03:31.265180 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265128 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-data\") pod \"31585445-5877-4947-8df5-f30317948127\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " Oct 31 00:03:31.265433 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265191 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-credential-socket\") pod \"02d621d9-9a83-45c7-bef8-9255f543c6f0\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " Oct 31 00:03:31.265433 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265219 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-podinfo\") pod \"02d621d9-9a83-45c7-bef8-9255f543c6f0\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " Oct 31 00:03:31.265433 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265255 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31585445-5877-4947-8df5-f30317948127-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31585445-5877-4947-8df5-f30317948127-istiod-ca-cert\") pod \"31585445-5877-4947-8df5-f30317948127\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " Oct 31 00:03:31.265433 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265309 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-certs\") pod \"02d621d9-9a83-45c7-bef8-9255f543c6f0\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " Oct 31 00:03:31.265433 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265361 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-data\") pod \"02d621d9-9a83-45c7-bef8-9255f543c6f0\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " Oct 31 00:03:31.265433 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265401 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-envoy\") pod \"31585445-5877-4947-8df5-f30317948127\" (UID: \"31585445-5877-4947-8df5-f30317948127\") " Oct 31 00:03:31.265941 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265524 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d621d9-9a83-45c7-bef8-9255f543c6f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d621d9-9a83-45c7-bef8-9255f543c6f0-istiod-ca-cert\") pod \"02d621d9-9a83-45c7-bef8-9255f543c6f0\" (UID: \"02d621d9-9a83-45c7-bef8-9255f543c6f0\") " Oct 31 00:03:31.265941 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265610 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-data" pod "31585445-5877-4947-8df5-f30317948127" (UID: "31585445-5877-4947-8df5-f30317948127"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:31.265941 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.265837 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-socket" pod "31585445-5877-4947-8df5-f30317948127" (UID: "31585445-5877-4947-8df5-f30317948127"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:31.266101 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.266009 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.266101 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.266029 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.266101 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.266045 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.266258 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.266179 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d621d9-9a83-45c7-bef8-9255f543c6f0-istiod-ca-cert" pod "02d621d9-9a83-45c7-bef8-9255f543c6f0" (UID: "02d621d9-9a83-45c7-bef8-9255f543c6f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:31.266389 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.266366 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-certs" pod "31585445-5877-4947-8df5-f30317948127" (UID: "31585445-5877-4947-8df5-f30317948127"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:31.266453 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.266387 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31585445-5877-4947-8df5-f30317948127-istiod-ca-cert" pod "31585445-5877-4947-8df5-f30317948127" (UID: "31585445-5877-4947-8df5-f30317948127"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:31.266737 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.266701 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-credential-socket" pod "31585445-5877-4947-8df5-f30317948127" (UID: "31585445-5877-4947-8df5-f30317948127"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:31.268424 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.267837 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-certs" pod "02d621d9-9a83-45c7-bef8-9255f543c6f0" (UID: "02d621d9-9a83-45c7-bef8-9255f543c6f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:31.269155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.268872 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31585445-5877-4947-8df5-f30317948127-istio-podinfo" pod "31585445-5877-4947-8df5-f30317948127" (UID: "31585445-5877-4947-8df5-f30317948127"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:03:31.269155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.268955 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-credential-socket" pod "02d621d9-9a83-45c7-bef8-9255f543c6f0" (UID: "02d621d9-9a83-45c7-bef8-9255f543c6f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:31.269155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.269033 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-envoy" pod "31585445-5877-4947-8df5-f30317948127" (UID: "31585445-5877-4947-8df5-f30317948127"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:31.269155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.269054 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-token" pod "02d621d9-9a83-45c7-bef8-9255f543c6f0" (UID: "02d621d9-9a83-45c7-bef8-9255f543c6f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:31.269155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.269133 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-data" pod "02d621d9-9a83-45c7-bef8-9255f543c6f0" (UID: "02d621d9-9a83-45c7-bef8-9255f543c6f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:31.269495 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.269261 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-kube-api-access-fssz2" pod "31585445-5877-4947-8df5-f30317948127" (UID: "31585445-5877-4947-8df5-f30317948127"). InnerVolumeSpecName "kube-api-access-fssz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:31.269495 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.269355 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-envoy" pod "02d621d9-9a83-45c7-bef8-9255f543c6f0" (UID: "02d621d9-9a83-45c7-bef8-9255f543c6f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:31.270346 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.270308 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-kube-api-access-2zqds" pod "02d621d9-9a83-45c7-bef8-9255f543c6f0" (UID: "02d621d9-9a83-45c7-bef8-9255f543c6f0"). InnerVolumeSpecName "kube-api-access-2zqds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:31.271029 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.271001 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-podinfo" pod "02d621d9-9a83-45c7-bef8-9255f543c6f0" (UID: "02d621d9-9a83-45c7-bef8-9255f543c6f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:03:31.271143 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.271128 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-istio-token" pod "31585445-5877-4947-8df5-f30317948127" (UID: "31585445-5877-4947-8df5-f30317948127"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:31.366646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366621 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fssz2\" (UniqueName: \"kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-kube-api-access-fssz2\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366645 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366657 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366665 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31585445-5877-4947-8df5-f30317948127-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366674 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366688 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366701 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366708 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d621d9-9a83-45c7-bef8-9255f543c6f0-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366716 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zqds\" (UniqueName: \"kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-kube-api-access-2zqds\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366724 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31585445-5877-4947-8df5-f30317948127-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366732 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31585445-5877-4947-8df5-f30317948127-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366739 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366747 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366756 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31585445-5877-4947-8df5-f30317948127-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.366773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.366773 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d621d9-9a83-45c7-bef8-9255f543c6f0-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:31.786096 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.786059 7570 generic.go:356] "Generic (PLEG): container finished" podID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerID="9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855" exitCode=0 Oct 31 00:03:31.786284 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.786122 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" Oct 31 00:03:31.786284 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.786152 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" event={"ID":"02d621d9-9a83-45c7-bef8-9255f543c6f0","Type":"ContainerDied","Data":"9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855"} Oct 31 00:03:31.786284 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.786218 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g" event={"ID":"02d621d9-9a83-45c7-bef8-9255f543c6f0","Type":"ContainerDied","Data":"d5212bf09fe33ededa07b37c5fd4c6875d2514a050be0030a4f0a7730fe5a5fb"} Oct 31 00:03:31.786284 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.786248 7570 scope.go:122] "RemoveContainer" containerID="9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855" Oct 31 00:03:31.788520 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.788451 7570 generic.go:356] "Generic (PLEG): container finished" podID="31585445-5877-4947-8df5-f30317948127" containerID="4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008" exitCode=0 Oct 31 00:03:31.788520 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.788510 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" event={"ID":"31585445-5877-4947-8df5-f30317948127","Type":"ContainerDied","Data":"4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008"} Oct 31 00:03:31.788728 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.788570 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" event={"ID":"31585445-5877-4947-8df5-f30317948127","Type":"ContainerDied","Data":"adcc5478966b2e686e68e2158a5ffab2f14af801021746655d1fb2477c487796"} Oct 31 00:03:31.788728 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.788521 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7" Oct 31 00:03:31.807379 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.807356 7570 scope.go:122] "RemoveContainer" containerID="9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855" Oct 31 00:03:31.807756 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:03:31.807731 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855\": container with ID starting with 9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855 not found: ID does not exist" containerID="9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855" Oct 31 00:03:31.807867 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.807763 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855"} err="failed to get container status \"9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855\": rpc error: code = NotFound desc = could not find container \"9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855\": container with ID starting with 9443dd1953512f800ad2d64d9e9b5f0394474848d401c2804b92ea6ebc898855 not found: ID does not exist" Oct 31 00:03:31.807867 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.807784 7570 scope.go:122] "RemoveContainer" containerID="4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008" Oct 31 00:03:31.820006 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.819983 7570 scope.go:122] "RemoveContainer" containerID="74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261" Oct 31 00:03:31.820342 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.820321 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7"] Oct 31 00:03:31.823449 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.823422 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-wg7d7"] Oct 31 00:03:31.830002 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.829985 7570 scope.go:122] "RemoveContainer" containerID="4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008" Oct 31 00:03:31.830268 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:03:31.830252 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008\": container with ID starting with 4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008 not found: ID does not exist" containerID="4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008" Oct 31 00:03:31.830311 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.830274 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008"} err="failed to get container status \"4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008\": rpc error: code = NotFound desc = could not find container \"4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008\": container with ID starting with 4576011e6ca43e121581cc549d941fae5125de8bfc114383960d103523553008 not found: ID does not exist" Oct 31 00:03:31.830311 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.830289 7570 scope.go:122] "RemoveContainer" containerID="74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261" Oct 31 00:03:31.830559 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:03:31.830543 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261\": container with ID starting with 74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261 not found: ID does not exist" containerID="74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261" Oct 31 00:03:31.830605 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.830563 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261"} err="failed to get container status \"74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261\": rpc error: code = NotFound desc = could not find container \"74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261\": container with ID starting with 74296406629c95b953feabc255c7e805eaf1696ab9c694b2caf632c995192261 not found: ID does not exist" Oct 31 00:03:31.839659 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.839635 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g"] Oct 31 00:03:31.843804 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.843783 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-t7b8g"] Oct 31 00:03:31.869357 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.869329 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j"] Oct 31 00:03:31.870092 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:31.870064 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dd3a15-sb95j"] Oct 31 00:03:32.603186 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:32.603145 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" path="/var/lib/kubelet/pods/02d621d9-9a83-45c7-bef8-9255f543c6f0/volumes" Oct 31 00:03:32.603742 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:32.603613 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31585445-5877-4947-8df5-f30317948127" path="/var/lib/kubelet/pods/31585445-5877-4947-8df5-f30317948127/volumes" Oct 31 00:03:32.603991 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:32.603973 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599a3946-97e4-439b-b2e7-1f1ce2eb6125" path="/var/lib/kubelet/pods/599a3946-97e4-439b-b2e7-1f1ce2eb6125/volumes" Oct 31 00:03:39.266256 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:39.266217 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx"] Oct 31 00:03:39.266803 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:39.266544 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" containerID="cri-o://f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626" gracePeriod=30 Oct 31 00:03:43.149656 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:43.149622 7570 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:03:43.150177 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:43.149688 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:03:44.608750 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.608722 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 31 00:03:44.723907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.723866 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570b6b11-715d-4a67-9360-78aa45580d57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570b6b11-715d-4a67-9360-78aa45580d57-istiod-ca-cert\") pod \"570b6b11-715d-4a67-9360-78aa45580d57\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " Oct 31 00:03:44.723907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.723922 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-socket\") pod \"570b6b11-715d-4a67-9360-78aa45580d57\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " Oct 31 00:03:44.724182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.723961 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-credential-socket\") pod \"570b6b11-715d-4a67-9360-78aa45580d57\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " Oct 31 00:03:44.724182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.724083 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-data\") pod \"570b6b11-715d-4a67-9360-78aa45580d57\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " Oct 31 00:03:44.724182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.724138 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-certs\") pod \"570b6b11-715d-4a67-9360-78aa45580d57\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " Oct 31 00:03:44.724182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.724177 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-envoy\") pod \"570b6b11-715d-4a67-9360-78aa45580d57\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " Oct 31 00:03:44.724407 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.724232 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-istio-token\" (UniqueName: \"kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-istio-token\") pod \"570b6b11-715d-4a67-9360-78aa45580d57\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " Oct 31 00:03:44.724407 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.724289 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/570b6b11-715d-4a67-9360-78aa45580d57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570b6b11-715d-4a67-9360-78aa45580d57-istio-podinfo\") pod \"570b6b11-715d-4a67-9360-78aa45580d57\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " Oct 31 00:03:44.724407 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.724319 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-kube-api-access-6td7j\" (UniqueName: \"kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-kube-api-access-6td7j\") pod \"570b6b11-715d-4a67-9360-78aa45580d57\" (UID: \"570b6b11-715d-4a67-9360-78aa45580d57\") " Oct 31 00:03:44.725184 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.725152 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-credential-socket" pod "570b6b11-715d-4a67-9360-78aa45580d57" (UID: "570b6b11-715d-4a67-9360-78aa45580d57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:44.725446 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.725420 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570b6b11-715d-4a67-9360-78aa45580d57-istiod-ca-cert" pod "570b6b11-715d-4a67-9360-78aa45580d57" (UID: "570b6b11-715d-4a67-9360-78aa45580d57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:44.725734 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.725700 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-socket" pod "570b6b11-715d-4a67-9360-78aa45580d57" (UID: "570b6b11-715d-4a67-9360-78aa45580d57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:44.726001 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.725976 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-certs" pod "570b6b11-715d-4a67-9360-78aa45580d57" (UID: "570b6b11-715d-4a67-9360-78aa45580d57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:44.726553 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.726523 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-data" pod "570b6b11-715d-4a67-9360-78aa45580d57" (UID: "570b6b11-715d-4a67-9360-78aa45580d57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:44.729548 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.729436 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-istio-token" pod "570b6b11-715d-4a67-9360-78aa45580d57" (UID: "570b6b11-715d-4a67-9360-78aa45580d57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:44.731606 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.731585 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/570b6b11-715d-4a67-9360-78aa45580d57-istio-podinfo" pod "570b6b11-715d-4a67-9360-78aa45580d57" (UID: "570b6b11-715d-4a67-9360-78aa45580d57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:03:44.733426 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.733404 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-envoy" pod "570b6b11-715d-4a67-9360-78aa45580d57" (UID: "570b6b11-715d-4a67-9360-78aa45580d57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:44.734083 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.734051 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-kube-api-access-6td7j" pod "570b6b11-715d-4a67-9360-78aa45580d57" (UID: "570b6b11-715d-4a67-9360-78aa45580d57"). InnerVolumeSpecName "kube-api-access-6td7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:44.827582 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.827329 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:44.827582 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.827375 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:44.827582 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.827391 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:44.827582 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.827405 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570b6b11-715d-4a67-9360-78aa45580d57-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:44.827582 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.827421 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6td7j\" (UniqueName: \"kubernetes.io/projected/570b6b11-715d-4a67-9360-78aa45580d57-kube-api-access-6td7j\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:44.827582 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.827436 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570b6b11-715d-4a67-9360-78aa45580d57-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:44.827582 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.827450 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:44.827582 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.827482 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:44.827582 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.827496 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570b6b11-715d-4a67-9360-78aa45580d57-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:03:44.879094 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.879062 7570 generic.go:356] "Generic (PLEG): container finished" podID="570b6b11-715d-4a67-9360-78aa45580d57" containerID="f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626" exitCode=0 Oct 31 00:03:44.879293 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.879139 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" Oct 31 00:03:44.879293 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.879158 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" event={"ID":"570b6b11-715d-4a67-9360-78aa45580d57","Type":"ContainerDied","Data":"f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626"} Oct 31 00:03:44.879293 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.879245 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx" event={"ID":"570b6b11-715d-4a67-9360-78aa45580d57","Type":"ContainerDied","Data":"41197724134ea2933938575acc176957120bf0d1f27a419c0c7a6e8711116922"} Oct 31 00:03:44.879293 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.879279 7570 scope.go:122] "RemoveContainer" containerID="f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626" Oct 31 00:03:44.894368 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.894343 7570 scope.go:122] "RemoveContainer" containerID="f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626" Oct 31 00:03:44.895231 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:03:44.894755 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626\": container with ID starting with f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626 not found: ID does not exist" containerID="f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626" Oct 31 00:03:44.895231 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.894790 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626"} err="failed to get container status \"f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626\": rpc error: code = NotFound desc = could not find container \"f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626\": container with ID starting with f5acf864833a324117f1446d88e788e2f73ecd412e43439ece7dd722cbcc9626 not found: ID does not exist" Oct 31 00:03:44.919124 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.918532 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx"] Oct 31 00:03:44.931342 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:44.931310 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-2p2qx"] Oct 31 00:03:46.601404 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:46.601373 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570b6b11-715d-4a67-9360-78aa45580d57" path="/var/lib/kubelet/pods/570b6b11-715d-4a67-9360-78aa45580d57/volumes" Oct 31 00:03:56.024356 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:56.024325 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm"] Oct 31 00:03:56.031294 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:56.031262 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-6tdnm"] Oct 31 00:03:56.601109 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:03:56.601078 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec6f9e3-832c-422c-8172-123b52f2e2e5" path="/var/lib/kubelet/pods/dec6f9e3-832c-422c-8172-123b52f2e2e5/volumes" Oct 31 00:03:59.411182 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:03:59.411144 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:04:04.010687 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.010656 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl"] Oct 31 00:04:04.011101 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011080 7570 cpu_manager.go:414] "containerMap: removing container" podUID="c53048d8-1746-42ae-91a4-f160a2948c62" containerName="revoke-keys" Oct 31 00:04:04.011101 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011091 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53048d8-1746-42ae-91a4-f160a2948c62" containerName="revoke-keys" Oct 31 00:04:04.011179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011109 7570 cpu_manager.go:414] "containerMap: removing container" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" Oct 31 00:04:04.011179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011115 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" Oct 31 00:04:04.011179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011136 7570 cpu_manager.go:414] "containerMap: removing container" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" Oct 31 00:04:04.011179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011142 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" Oct 31 00:04:04.011179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011150 7570 cpu_manager.go:414] "containerMap: removing container" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" Oct 31 00:04:04.011179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011154 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" Oct 31 00:04:04.011179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011167 7570 cpu_manager.go:414] "containerMap: removing container" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" Oct 31 00:04:04.011179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011171 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" Oct 31 00:04:04.011179 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011182 7570 cpu_manager.go:414] "containerMap: removing container" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" Oct 31 00:04:04.011444 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011187 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" Oct 31 00:04:04.011444 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011243 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" Oct 31 00:04:04.011444 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011252 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="570b6b11-715d-4a67-9360-78aa45580d57" containerName="istio-proxy" Oct 31 00:04:04.011444 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011263 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53048d8-1746-42ae-91a4-f160a2948c62" containerName="revoke-keys" Oct 31 00:04:04.011444 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011278 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="809bef14-259b-4a21-b170-a6a701e00e6c" containerName="main" Oct 31 00:04:04.011444 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.011288 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d621d9-9a83-45c7-bef8-9255f543c6f0" containerName="istio-proxy" Oct 31 00:04:04.015896 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.015880 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:04.021169 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.021135 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-k4qsl\" (UID: \"97ac295b-1237-47ff-891f-f7974c867c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:04.021312 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.021184 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmpc\" (UniqueName: \"kubernetes.io/projected/97ac295b-1237-47ff-891f-f7974c867c2e-kube-api-access-rtmpc\") pod \"maas-api-revoke-sub-85237e45-k4qsl\" (UID: \"97ac295b-1237-47ff-891f-f7974c867c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:04.021312 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.021214 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-k4qsl\" (UID: \"97ac295b-1237-47ff-891f-f7974c867c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:04.024855 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.024318 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl"] Oct 31 00:04:04.026420 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.026393 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:04:04.026794 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.026774 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:04:04.027035 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.027018 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:04:04.030110 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.027231 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:04:04.122161 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.122123 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-k4qsl\" (UID: \"97ac295b-1237-47ff-891f-f7974c867c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:04.122161 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.122167 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmpc\" (UniqueName: \"kubernetes.io/projected/97ac295b-1237-47ff-891f-f7974c867c2e-kube-api-access-rtmpc\") pod \"maas-api-revoke-sub-85237e45-k4qsl\" (UID: \"97ac295b-1237-47ff-891f-f7974c867c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:04.122421 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.122194 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-k4qsl\" (UID: \"97ac295b-1237-47ff-891f-f7974c867c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:04.122801 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.122780 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-k4qsl\" (UID: \"97ac295b-1237-47ff-891f-f7974c867c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:04.124762 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.124742 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-k4qsl\" (UID: \"97ac295b-1237-47ff-891f-f7974c867c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:04.134739 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.134719 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmpc\" (UniqueName: \"kubernetes.io/projected/97ac295b-1237-47ff-891f-f7974c867c2e-kube-api-access-rtmpc\") pod \"maas-api-revoke-sub-85237e45-k4qsl\" (UID: \"97ac295b-1237-47ff-891f-f7974c867c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:04.334581 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.334552 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:04.611410 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:04.611379 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl"] Oct 31 00:04:04.612802 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:04:04.612777 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ac295b_1237_47ff_891f_f7974c867c2e.slice/crio-6eb5e17b1078c469996a62108976448e21afafc5a96406b45a895ce5bf75f2b9 WatchSource:0}: Error finding container 6eb5e17b1078c469996a62108976448e21afafc5a96406b45a895ce5bf75f2b9: Status 404 returned error can't find the container with id 6eb5e17b1078c469996a62108976448e21afafc5a96406b45a895ce5bf75f2b9 Oct 31 00:04:05.015084 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:05.015047 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" event={"ID":"97ac295b-1237-47ff-891f-f7974c867c2e","Type":"ContainerStarted","Data":"0ed8ee7940f11553dca12bc017f6736a26015c04176ca5f929ee75f2244092a0"} Oct 31 00:04:05.015603 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:05.015103 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" event={"ID":"97ac295b-1237-47ff-891f-f7974c867c2e","Type":"ContainerStarted","Data":"6eb5e17b1078c469996a62108976448e21afafc5a96406b45a895ce5bf75f2b9"} Oct 31 00:04:05.033738 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:05.033695 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" podStartSLOduration=2.033682919 podStartE2EDuration="2.033682919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:05.032710605 +0000 UTC m=+2656.551711697" watchObservedRunningTime="2026-10-31 00:04:05.033682919 +0000 UTC m=+2656.552684008" Oct 31 00:04:06.023294 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:06.023249 7570 generic.go:356] "Generic (PLEG): container finished" podID="97ac295b-1237-47ff-891f-f7974c867c2e" containerID="0ed8ee7940f11553dca12bc017f6736a26015c04176ca5f929ee75f2244092a0" exitCode=0 Oct 31 00:04:06.023725 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:06.023338 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" event={"ID":"97ac295b-1237-47ff-891f-f7974c867c2e","Type":"ContainerDied","Data":"0ed8ee7940f11553dca12bc017f6736a26015c04176ca5f929ee75f2244092a0"} Oct 31 00:04:07.166992 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:07.166969 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:07.244044 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:07.244014 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ac295b-1237-47ff-891f-f7974c867c2e-kube-api-access-rtmpc\" (UniqueName: \"kubernetes.io/projected/97ac295b-1237-47ff-891f-f7974c867c2e-kube-api-access-rtmpc\") pod \"97ac295b-1237-47ff-891f-f7974c867c2e\" (UID: \"97ac295b-1237-47ff-891f-f7974c867c2e\") " Oct 31 00:04:07.244212 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:07.244077 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-xks-ca\") pod \"97ac295b-1237-47ff-891f-f7974c867c2e\" (UID: \"97ac295b-1237-47ff-891f-f7974c867c2e\") " Oct 31 00:04:07.244212 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:07.244150 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-openshift-ca\") pod \"97ac295b-1237-47ff-891f-f7974c867c2e\" (UID: \"97ac295b-1237-47ff-891f-f7974c867c2e\") " Oct 31 00:04:07.244526 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:07.244504 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-openshift-ca" pod "97ac295b-1237-47ff-891f-f7974c867c2e" (UID: "97ac295b-1237-47ff-891f-f7974c867c2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:04:07.246450 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:07.246420 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-xks-ca" pod "97ac295b-1237-47ff-891f-f7974c867c2e" (UID: "97ac295b-1237-47ff-891f-f7974c867c2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:07.246450 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:07.246441 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ac295b-1237-47ff-891f-f7974c867c2e-kube-api-access-rtmpc" pod "97ac295b-1237-47ff-891f-f7974c867c2e" (UID: "97ac295b-1237-47ff-891f-f7974c867c2e"). InnerVolumeSpecName "kube-api-access-rtmpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:04:07.345571 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:07.345541 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:07.345571 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:07.345567 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ac295b-1237-47ff-891f-f7974c867c2e-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:07.345762 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:07.345578 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmpc\" (UniqueName: \"kubernetes.io/projected/97ac295b-1237-47ff-891f-f7974c867c2e-kube-api-access-rtmpc\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:08.036931 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:08.036894 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" Oct 31 00:04:08.037105 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:08.036935 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl" event={"ID":"97ac295b-1237-47ff-891f-f7974c867c2e","Type":"ContainerDied","Data":"6eb5e17b1078c469996a62108976448e21afafc5a96406b45a895ce5bf75f2b9"} Oct 31 00:04:08.037105 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:08.036984 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb5e17b1078c469996a62108976448e21afafc5a96406b45a895ce5bf75f2b9" Oct 31 00:04:11.032842 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.032810 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp"] Oct 31 00:04:11.033278 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.033266 7570 cpu_manager.go:414] "containerMap: removing container" podUID="97ac295b-1237-47ff-891f-f7974c867c2e" containerName="revoke-keys" Oct 31 00:04:11.033317 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.033280 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ac295b-1237-47ff-891f-f7974c867c2e" containerName="revoke-keys" Oct 31 00:04:11.033365 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.033356 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="31585445-5877-4947-8df5-f30317948127" containerName="istio-proxy" Oct 31 00:04:11.033398 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.033371 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ac295b-1237-47ff-891f-f7974c867c2e" containerName="revoke-keys" Oct 31 00:04:11.036847 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.036830 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.078425 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.078392 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.078610 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.078431 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.078610 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.078535 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.078703 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.078619 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.078703 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.078657 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.078703 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.078683 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.078823 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.078780 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5czz\" (UniqueName: \"kubernetes.io/projected/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-kube-api-access-p5czz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.078823 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.078815 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.078900 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.078852 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.084756 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.084721 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp"] Oct 31 00:04:11.180366 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.180332 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.180366 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.180373 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.180646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.180393 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5czz\" (UniqueName: \"kubernetes.io/projected/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-kube-api-access-p5czz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.180646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.180418 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.180646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.180450 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.180646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.180484 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.180646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.180503 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.180646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.180534 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.180646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.180591 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.180988 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.180826 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.180988 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.180871 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.180988 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.180941 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.181125 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.181062 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.181170 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.181146 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.183052 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.183029 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.183161 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.183134 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.192985 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.192964 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.193078 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.192996 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5czz\" (UniqueName: \"kubernetes.io/projected/229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d-kube-api-access-p5czz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-g4dbp\" (UID: \"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.351488 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.351424 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:11.635251 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.635224 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp"] Oct 31 00:04:11.636737 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:04:11.636706 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229ef7e0_5e24_4f6c_b60d_e1e0bf2dc32d.slice/crio-8f3e2021cd395bf8ce88401d384cec8db3295232dc3f74f7b04101e1d7b441fa WatchSource:0}: Error finding container 8f3e2021cd395bf8ce88401d384cec8db3295232dc3f74f7b04101e1d7b441fa: Status 404 returned error can't find the container with id 8f3e2021cd395bf8ce88401d384cec8db3295232dc3f74f7b04101e1d7b441fa Oct 31 00:04:11.638841 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.638811 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:04:11.638912 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.638877 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:04:11.638948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:11.638911 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:04:12.062534 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:12.062428 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" event={"ID":"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d","Type":"ContainerStarted","Data":"0d032e03ce370bcd9ab50e7f2a19c32c2bff433e2e8b38247b6eecbdbf216218"} Oct 31 00:04:12.062534 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:12.062512 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" event={"ID":"229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d","Type":"ContainerStarted","Data":"8f3e2021cd395bf8ce88401d384cec8db3295232dc3f74f7b04101e1d7b441fa"} Oct 31 00:04:12.100487 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:12.092913 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" podStartSLOduration=2.092894533 podStartE2EDuration="2.092894533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:12.090190883 +0000 UTC m=+2663.609191973" watchObservedRunningTime="2026-10-31 00:04:12.092894533 +0000 UTC m=+2663.611895624" Oct 31 00:04:12.352517 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:12.352487 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:12.354121 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:12.354097 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g4dbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:04:12.354249 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:12.354152 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" podUID="229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:04:13.352500 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:13.352437 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g4dbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:04:13.353009 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:13.352540 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" podUID="229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:04:14.352071 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:14.352035 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g4dbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:04:14.352279 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:14.352099 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" podUID="229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:04:15.352146 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:15.352116 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g4dbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:04:15.352579 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:15.352170 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" podUID="229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:04:16.352253 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:16.352217 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g4dbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:04:16.352714 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:16.352286 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" podUID="229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:04:17.352193 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:17.352155 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g4dbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:04:17.352363 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:17.352219 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" podUID="229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:04:18.352887 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:18.352848 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g4dbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:04:18.353311 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:18.352917 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" podUID="229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:04:19.887717 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:19.887683 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:19.888236 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:19.888145 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:19.888894 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:19.888876 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" Oct 31 00:04:29.413965 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:04:29.413925 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:04:32.024369 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:32.024334 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp"] Oct 31 00:04:32.029638 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:32.029608 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88aa7a16-47cqp"] Oct 31 00:04:32.601119 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:32.601086 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfb49dc-17f9-47ea-a294-246091178077" path="/var/lib/kubelet/pods/8cfb49dc-17f9-47ea-a294-246091178077/volumes" Oct 31 00:04:41.027719 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:41.027685 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q"] Oct 31 00:04:41.030297 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:41.030276 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ca4c26-2hx6q"] Oct 31 00:04:42.601284 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:42.601251 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6530eea0-8758-41fb-8249-a151d46c8ebf" path="/var/lib/kubelet/pods/6530eea0-8758-41fb-8249-a151d46c8ebf/volumes" Oct 31 00:04:43.798907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:43.798876 7570 scope.go:122] "RemoveContainer" containerID="c227a0741ae0910076f4793372f145e326dc30a40e2f82fb6577a0ab4f499609" Oct 31 00:04:55.013484 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.013431 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9"] Oct 31 00:04:55.016802 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.016788 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:55.023182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.023156 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:04:55.023363 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.023167 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:04:55.023482 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.023300 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:04:55.023695 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.023679 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:04:55.026982 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.026961 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9"] Oct 31 00:04:55.101354 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.101325 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kh7d\" (UniqueName: \"kubernetes.io/projected/7abfd60b-5291-4a6d-85e9-e7aeba227819-kube-api-access-5kh7d\") pod \"maas-api-revoke-sub-c1e271a9-95md9\" (UID: \"7abfd60b-5291-4a6d-85e9-e7aeba227819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:55.101516 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.101388 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-95md9\" (UID: \"7abfd60b-5291-4a6d-85e9-e7aeba227819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:55.101563 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.101535 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-95md9\" (UID: \"7abfd60b-5291-4a6d-85e9-e7aeba227819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:55.202803 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.202775 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-95md9\" (UID: \"7abfd60b-5291-4a6d-85e9-e7aeba227819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:55.202933 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.202821 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kh7d\" (UniqueName: \"kubernetes.io/projected/7abfd60b-5291-4a6d-85e9-e7aeba227819-kube-api-access-5kh7d\") pod \"maas-api-revoke-sub-c1e271a9-95md9\" (UID: \"7abfd60b-5291-4a6d-85e9-e7aeba227819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:55.202933 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.202864 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-95md9\" (UID: \"7abfd60b-5291-4a6d-85e9-e7aeba227819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:55.203508 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.203490 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-95md9\" (UID: \"7abfd60b-5291-4a6d-85e9-e7aeba227819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:55.205395 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.205375 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-95md9\" (UID: \"7abfd60b-5291-4a6d-85e9-e7aeba227819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:55.215600 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.215575 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kh7d\" (UniqueName: \"kubernetes.io/projected/7abfd60b-5291-4a6d-85e9-e7aeba227819-kube-api-access-5kh7d\") pod \"maas-api-revoke-sub-c1e271a9-95md9\" (UID: \"7abfd60b-5291-4a6d-85e9-e7aeba227819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:55.333147 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.333122 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:55.613834 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:55.613812 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9"] Oct 31 00:04:55.616687 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:04:55.616662 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abfd60b_5291_4a6d_85e9_e7aeba227819.slice/crio-e02c9fa4f89ee5dae2d90794697852b0be6eead53bd8202e3035620e8b7c4f46 WatchSource:0}: Error finding container e02c9fa4f89ee5dae2d90794697852b0be6eead53bd8202e3035620e8b7c4f46: Status 404 returned error can't find the container with id e02c9fa4f89ee5dae2d90794697852b0be6eead53bd8202e3035620e8b7c4f46 Oct 31 00:04:56.391542 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:56.391508 7570 generic.go:356] "Generic (PLEG): container finished" podID="7abfd60b-5291-4a6d-85e9-e7aeba227819" containerID="16c3f4f57408294dae89f5e3e188ffe951047b5e1d15ff4626926e622be7ab91" exitCode=0 Oct 31 00:04:56.391950 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:56.391595 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" event={"ID":"7abfd60b-5291-4a6d-85e9-e7aeba227819","Type":"ContainerDied","Data":"16c3f4f57408294dae89f5e3e188ffe951047b5e1d15ff4626926e622be7ab91"} Oct 31 00:04:56.391950 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:56.391669 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" event={"ID":"7abfd60b-5291-4a6d-85e9-e7aeba227819","Type":"ContainerStarted","Data":"e02c9fa4f89ee5dae2d90794697852b0be6eead53bd8202e3035620e8b7c4f46"} Oct 31 00:04:57.539561 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:57.539538 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:57.627062 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:57.627030 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-openshift-ca\") pod \"7abfd60b-5291-4a6d-85e9-e7aeba227819\" (UID: \"7abfd60b-5291-4a6d-85e9-e7aeba227819\") " Oct 31 00:04:57.627241 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:57.627148 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abfd60b-5291-4a6d-85e9-e7aeba227819-kube-api-access-5kh7d\" (UniqueName: \"kubernetes.io/projected/7abfd60b-5291-4a6d-85e9-e7aeba227819-kube-api-access-5kh7d\") pod \"7abfd60b-5291-4a6d-85e9-e7aeba227819\" (UID: \"7abfd60b-5291-4a6d-85e9-e7aeba227819\") " Oct 31 00:04:57.627310 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:57.627237 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-xks-ca\") pod \"7abfd60b-5291-4a6d-85e9-e7aeba227819\" (UID: \"7abfd60b-5291-4a6d-85e9-e7aeba227819\") " Oct 31 00:04:57.627487 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:57.627434 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-openshift-ca" pod "7abfd60b-5291-4a6d-85e9-e7aeba227819" (UID: "7abfd60b-5291-4a6d-85e9-e7aeba227819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:04:57.627728 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:57.627702 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:57.629409 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:57.629386 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-xks-ca" pod "7abfd60b-5291-4a6d-85e9-e7aeba227819" (UID: "7abfd60b-5291-4a6d-85e9-e7aeba227819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:57.629595 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:57.629580 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abfd60b-5291-4a6d-85e9-e7aeba227819-kube-api-access-5kh7d" pod "7abfd60b-5291-4a6d-85e9-e7aeba227819" (UID: "7abfd60b-5291-4a6d-85e9-e7aeba227819"). InnerVolumeSpecName "kube-api-access-5kh7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:04:57.728804 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:57.728731 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kh7d\" (UniqueName: \"kubernetes.io/projected/7abfd60b-5291-4a6d-85e9-e7aeba227819-kube-api-access-5kh7d\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:57.728804 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:57.728760 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abfd60b-5291-4a6d-85e9-e7aeba227819-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:58.405413 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:58.405384 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" Oct 31 00:04:58.405610 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:58.405429 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9" event={"ID":"7abfd60b-5291-4a6d-85e9-e7aeba227819","Type":"ContainerDied","Data":"e02c9fa4f89ee5dae2d90794697852b0be6eead53bd8202e3035620e8b7c4f46"} Oct 31 00:04:58.405610 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:04:58.405496 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e02c9fa4f89ee5dae2d90794697852b0be6eead53bd8202e3035620e8b7c4f46" Oct 31 00:04:59.412759 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:04:59.412718 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:05:11.017971 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.017935 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs"] Oct 31 00:05:11.018591 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.018127 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" podUID="363febe9-9102-48c9-8faa-09bf49151bf0" containerName="discovery" containerID="cri-o://5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da" gracePeriod=30 Oct 31 00:05:11.270803 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.270747 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 31 00:05:11.383389 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.383356 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-cacerts\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-cacerts\") pod \"363febe9-9102-48c9-8faa-09bf49151bf0\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " Oct 31 00:05:11.383586 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.383401 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-ca-configmap\") pod \"363febe9-9102-48c9-8faa-09bf49151bf0\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " Oct 31 00:05:11.383586 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.383532 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-dns-cert\") pod \"363febe9-9102-48c9-8faa-09bf49151bf0\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " Oct 31 00:05:11.383586 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.383567 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363febe9-9102-48c9-8faa-09bf49151bf0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/363febe9-9102-48c9-8faa-09bf49151bf0-local-certs\") pod \"363febe9-9102-48c9-8faa-09bf49151bf0\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " Oct 31 00:05:11.383767 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.383663 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-kubeconfig\") pod \"363febe9-9102-48c9-8faa-09bf49151bf0\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " Oct 31 00:05:11.383767 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.383704 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-kube-api-access-fhsj4\" (UniqueName: \"kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-kube-api-access-fhsj4\") pod \"363febe9-9102-48c9-8faa-09bf49151bf0\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " Oct 31 00:05:11.383767 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.383735 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-istio-token\" (UniqueName: \"kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-istio-token\") pod \"363febe9-9102-48c9-8faa-09bf49151bf0\" (UID: \"363febe9-9102-48c9-8faa-09bf49151bf0\") " Oct 31 00:05:11.383926 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.383873 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-ca-configmap" pod "363febe9-9102-48c9-8faa-09bf49151bf0" (UID: "363febe9-9102-48c9-8faa-09bf49151bf0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:05:11.384165 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.384138 7570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-ca-configmap\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:11.385970 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.385937 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363febe9-9102-48c9-8faa-09bf49151bf0-local-certs" pod "363febe9-9102-48c9-8faa-09bf49151bf0" (UID: "363febe9-9102-48c9-8faa-09bf49151bf0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:05:11.386632 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.386605 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-cacerts" pod "363febe9-9102-48c9-8faa-09bf49151bf0" (UID: "363febe9-9102-48c9-8faa-09bf49151bf0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:05:11.386747 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.386624 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-kube-api-access-fhsj4" pod "363febe9-9102-48c9-8faa-09bf49151bf0" (UID: "363febe9-9102-48c9-8faa-09bf49151bf0"). InnerVolumeSpecName "kube-api-access-fhsj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:11.386747 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.386643 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-kubeconfig" pod "363febe9-9102-48c9-8faa-09bf49151bf0" (UID: "363febe9-9102-48c9-8faa-09bf49151bf0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:05:11.386747 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.386658 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-dns-cert" pod "363febe9-9102-48c9-8faa-09bf49151bf0" (UID: "363febe9-9102-48c9-8faa-09bf49151bf0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:05:11.386856 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.386770 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-istio-token" pod "363febe9-9102-48c9-8faa-09bf49151bf0" (UID: "363febe9-9102-48c9-8faa-09bf49151bf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:11.484710 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.484680 7570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-kubeconfig\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:11.484710 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.484707 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhsj4\" (UniqueName: \"kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-kube-api-access-fhsj4\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:11.484875 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.484720 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363febe9-9102-48c9-8faa-09bf49151bf0-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:11.484875 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.484730 7570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-cacerts\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:11.484875 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.484739 7570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/363febe9-9102-48c9-8faa-09bf49151bf0-istio-csr-dns-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:11.484875 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.484747 7570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/363febe9-9102-48c9-8faa-09bf49151bf0-local-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:11.561983 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.561898 7570 generic.go:356] "Generic (PLEG): container finished" podID="363febe9-9102-48c9-8faa-09bf49151bf0" containerID="5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da" exitCode=0 Oct 31 00:05:11.561983 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.561945 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" Oct 31 00:05:11.562208 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.561987 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" event={"ID":"363febe9-9102-48c9-8faa-09bf49151bf0","Type":"ContainerDied","Data":"5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da"} Oct 31 00:05:11.562208 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.562051 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs" event={"ID":"363febe9-9102-48c9-8faa-09bf49151bf0","Type":"ContainerDied","Data":"6fce487599129fc1d9adaa23349421c3b96107e00457c968c9b1e584128436ea"} Oct 31 00:05:11.562208 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.562079 7570 scope.go:122] "RemoveContainer" containerID="5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da" Oct 31 00:05:11.576578 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.576565 7570 scope.go:122] "RemoveContainer" containerID="5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da" Oct 31 00:05:11.577068 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:05:11.577049 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da\": container with ID starting with 5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da not found: ID does not exist" containerID="5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da" Oct 31 00:05:11.577130 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.577076 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da"} err="failed to get container status \"5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da\": rpc error: code = NotFound desc = could not find container \"5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da\": container with ID starting with 5baa4df5310385e3528f1658be637071638b4b59eef536e839628cb79e4fb5da not found: ID does not exist" Oct 31 00:05:11.588715 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.588693 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs"] Oct 31 00:05:11.596878 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:11.596859 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mxggs"] Oct 31 00:05:12.600428 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:12.600399 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363febe9-9102-48c9-8faa-09bf49151bf0" path="/var/lib/kubelet/pods/363febe9-9102-48c9-8faa-09bf49151bf0/volumes" Oct 31 00:05:13.023677 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.023580 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq"] Oct 31 00:05:13.026361 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.026333 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d85a55-vs5vq"] Oct 31 00:05:13.836424 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.836345 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb"] Oct 31 00:05:13.836842 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.836827 7570 cpu_manager.go:414] "containerMap: removing container" podUID="7abfd60b-5291-4a6d-85e9-e7aeba227819" containerName="revoke-keys" Oct 31 00:05:13.836886 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.836842 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abfd60b-5291-4a6d-85e9-e7aeba227819" containerName="revoke-keys" Oct 31 00:05:13.836886 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.836862 7570 cpu_manager.go:414] "containerMap: removing container" podUID="363febe9-9102-48c9-8faa-09bf49151bf0" containerName="discovery" Oct 31 00:05:13.836886 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.836867 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363febe9-9102-48c9-8faa-09bf49151bf0" containerName="discovery" Oct 31 00:05:13.836974 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.836939 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abfd60b-5291-4a6d-85e9-e7aeba227819" containerName="revoke-keys" Oct 31 00:05:13.836974 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.836952 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="363febe9-9102-48c9-8faa-09bf49151bf0" containerName="discovery" Oct 31 00:05:13.841482 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.841446 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:13.847479 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.846775 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:05:13.847479 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.846842 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:05:13.847479 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.846904 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:05:13.847479 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.846988 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:05:13.850903 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.850881 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb"] Oct 31 00:05:13.908581 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.908557 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-l6jxb\" (UID: \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:13.908719 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.908696 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sft4p\" (UniqueName: \"kubernetes.io/projected/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-kube-api-access-sft4p\") pod \"maas-api-revoke-sub-2e1f5b5f-l6jxb\" (UID: \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:13.908785 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:13.908771 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-l6jxb\" (UID: \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:14.009980 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:14.009952 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-l6jxb\" (UID: \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:14.010108 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:14.009998 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-l6jxb\" (UID: \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:14.010108 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:14.010101 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sft4p\" (UniqueName: \"kubernetes.io/projected/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-kube-api-access-sft4p\") pod \"maas-api-revoke-sub-2e1f5b5f-l6jxb\" (UID: \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:14.010595 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:14.010573 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-l6jxb\" (UID: \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:14.012589 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:14.012571 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-l6jxb\" (UID: \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:14.022648 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:14.022624 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sft4p\" (UniqueName: \"kubernetes.io/projected/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-kube-api-access-sft4p\") pod \"maas-api-revoke-sub-2e1f5b5f-l6jxb\" (UID: \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:14.157157 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:14.157084 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:14.438501 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:14.438450 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb"] Oct 31 00:05:14.439683 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:05:14.439654 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a79d5c6_7859_4f4c_a8d4_5713eb8185ac.slice/crio-89eba1dbf0398079da40ff53f1fbf2cc9b84b31fd9e3871f39a29901767b6377 WatchSource:0}: Error finding container 89eba1dbf0398079da40ff53f1fbf2cc9b84b31fd9e3871f39a29901767b6377: Status 404 returned error can't find the container with id 89eba1dbf0398079da40ff53f1fbf2cc9b84b31fd9e3871f39a29901767b6377 Oct 31 00:05:14.583701 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:14.583661 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" event={"ID":"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac","Type":"ContainerStarted","Data":"89eba1dbf0398079da40ff53f1fbf2cc9b84b31fd9e3871f39a29901767b6377"} Oct 31 00:05:14.601029 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:14.601002 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad2904e-63ba-45d4-b7dd-60f0edd67962" path="/var/lib/kubelet/pods/2ad2904e-63ba-45d4-b7dd-60f0edd67962/volumes" Oct 31 00:05:15.591889 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:15.591859 7570 generic.go:356] "Generic (PLEG): container finished" podID="5a79d5c6-7859-4f4c-a8d4-5713eb8185ac" containerID="2b4189433db6c405d30bf499a00e39c25d372dba454dcc31c0818aa2af8d554a" exitCode=0 Oct 31 00:05:15.592261 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:15.591943 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" event={"ID":"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac","Type":"ContainerDied","Data":"2b4189433db6c405d30bf499a00e39c25d372dba454dcc31c0818aa2af8d554a"} Oct 31 00:05:16.732819 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:16.732794 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:16.837723 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:16.837694 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-xks-ca\") pod \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\" (UID: \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\") " Oct 31 00:05:16.837866 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:16.837747 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-openshift-ca\") pod \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\" (UID: \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\") " Oct 31 00:05:16.837866 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:16.837837 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-kube-api-access-sft4p\" (UniqueName: \"kubernetes.io/projected/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-kube-api-access-sft4p\") pod \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\" (UID: \"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac\") " Oct 31 00:05:16.838100 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:16.838077 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-openshift-ca" pod "5a79d5c6-7859-4f4c-a8d4-5713eb8185ac" (UID: "5a79d5c6-7859-4f4c-a8d4-5713eb8185ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:05:16.840039 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:16.840006 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-kube-api-access-sft4p" pod "5a79d5c6-7859-4f4c-a8d4-5713eb8185ac" (UID: "5a79d5c6-7859-4f4c-a8d4-5713eb8185ac"). InnerVolumeSpecName "kube-api-access-sft4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:16.840121 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:16.840013 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-xks-ca" pod "5a79d5c6-7859-4f4c-a8d4-5713eb8185ac" (UID: "5a79d5c6-7859-4f4c-a8d4-5713eb8185ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:05:16.939260 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:16.939202 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:16.939260 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:16.939225 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sft4p\" (UniqueName: \"kubernetes.io/projected/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-kube-api-access-sft4p\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:16.939260 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:16.939235 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:17.608046 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:17.608011 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" Oct 31 00:05:17.608046 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:17.608043 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb" event={"ID":"5a79d5c6-7859-4f4c-a8d4-5713eb8185ac","Type":"ContainerDied","Data":"89eba1dbf0398079da40ff53f1fbf2cc9b84b31fd9e3871f39a29901767b6377"} Oct 31 00:05:17.608258 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:17.608085 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89eba1dbf0398079da40ff53f1fbf2cc9b84b31fd9e3871f39a29901767b6377" Oct 31 00:05:24.743564 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.743534 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z"] Oct 31 00:05:24.744204 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.744184 7570 cpu_manager.go:414] "containerMap: removing container" podUID="5a79d5c6-7859-4f4c-a8d4-5713eb8185ac" containerName="revoke-keys" Oct 31 00:05:24.744299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.744207 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a79d5c6-7859-4f4c-a8d4-5713eb8185ac" containerName="revoke-keys" Oct 31 00:05:24.744354 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.744317 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a79d5c6-7859-4f4c-a8d4-5713eb8185ac" containerName="revoke-keys" Oct 31 00:05:24.747751 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.747731 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:24.755532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.755503 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:05:24.755667 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.755574 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:05:24.757040 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.756624 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z"] Oct 31 00:05:24.757358 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.757338 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:05:24.757580 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.757564 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:05:24.917432 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.917401 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9pc8z\" (UID: \"63eddb85-3779-4cf7-8a98-2e51939e7639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:24.917599 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.917450 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9pc8z\" (UID: \"63eddb85-3779-4cf7-8a98-2e51939e7639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:24.917658 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:24.917619 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpsld\" (UniqueName: \"kubernetes.io/projected/63eddb85-3779-4cf7-8a98-2e51939e7639-kube-api-access-wpsld\") pod \"maas-api-revoke-sub-ec843018-9pc8z\" (UID: \"63eddb85-3779-4cf7-8a98-2e51939e7639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:25.018798 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:25.018704 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpsld\" (UniqueName: \"kubernetes.io/projected/63eddb85-3779-4cf7-8a98-2e51939e7639-kube-api-access-wpsld\") pod \"maas-api-revoke-sub-ec843018-9pc8z\" (UID: \"63eddb85-3779-4cf7-8a98-2e51939e7639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:25.018798 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:25.018754 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9pc8z\" (UID: \"63eddb85-3779-4cf7-8a98-2e51939e7639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:25.018798 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:25.018797 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9pc8z\" (UID: \"63eddb85-3779-4cf7-8a98-2e51939e7639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:25.019498 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:25.019449 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9pc8z\" (UID: \"63eddb85-3779-4cf7-8a98-2e51939e7639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:25.021321 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:25.021299 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9pc8z\" (UID: \"63eddb85-3779-4cf7-8a98-2e51939e7639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:25.031696 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:25.031674 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpsld\" (UniqueName: \"kubernetes.io/projected/63eddb85-3779-4cf7-8a98-2e51939e7639-kube-api-access-wpsld\") pod \"maas-api-revoke-sub-ec843018-9pc8z\" (UID: \"63eddb85-3779-4cf7-8a98-2e51939e7639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:25.064933 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:25.064911 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:25.342656 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:25.342632 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z"] Oct 31 00:05:25.345112 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:05:25.345075 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63eddb85_3779_4cf7_8a98_2e51939e7639.slice/crio-a202a474d452e131e6c99fa42bc96d98c753898f1a034174f921e236a23a1c33 WatchSource:0}: Error finding container a202a474d452e131e6c99fa42bc96d98c753898f1a034174f921e236a23a1c33: Status 404 returned error can't find the container with id a202a474d452e131e6c99fa42bc96d98c753898f1a034174f921e236a23a1c33 Oct 31 00:05:25.657029 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:25.656996 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" event={"ID":"63eddb85-3779-4cf7-8a98-2e51939e7639","Type":"ContainerStarted","Data":"9510c9ff322634d2aadb923a23f0689fda9cb643ac41059d56abf72792fb65a0"} Oct 31 00:05:25.657166 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:25.657085 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" event={"ID":"63eddb85-3779-4cf7-8a98-2e51939e7639","Type":"ContainerStarted","Data":"a202a474d452e131e6c99fa42bc96d98c753898f1a034174f921e236a23a1c33"} Oct 31 00:05:26.664387 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:26.664348 7570 generic.go:356] "Generic (PLEG): container finished" podID="63eddb85-3779-4cf7-8a98-2e51939e7639" containerID="9510c9ff322634d2aadb923a23f0689fda9cb643ac41059d56abf72792fb65a0" exitCode=0 Oct 31 00:05:26.664793 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:26.664442 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" event={"ID":"63eddb85-3779-4cf7-8a98-2e51939e7639","Type":"ContainerDied","Data":"9510c9ff322634d2aadb923a23f0689fda9cb643ac41059d56abf72792fb65a0"} Oct 31 00:05:27.825272 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:27.825243 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:27.950938 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:27.950834 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-openshift-ca\") pod \"63eddb85-3779-4cf7-8a98-2e51939e7639\" (UID: \"63eddb85-3779-4cf7-8a98-2e51939e7639\") " Oct 31 00:05:27.953418 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:27.953392 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63eddb85-3779-4cf7-8a98-2e51939e7639-kube-api-access-wpsld\" (UniqueName: \"kubernetes.io/projected/63eddb85-3779-4cf7-8a98-2e51939e7639-kube-api-access-wpsld\") pod \"63eddb85-3779-4cf7-8a98-2e51939e7639\" (UID: \"63eddb85-3779-4cf7-8a98-2e51939e7639\") " Oct 31 00:05:27.953720 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:27.953700 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-xks-ca\") pod \"63eddb85-3779-4cf7-8a98-2e51939e7639\" (UID: \"63eddb85-3779-4cf7-8a98-2e51939e7639\") " Oct 31 00:05:27.959495 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:27.951540 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-openshift-ca" pod "63eddb85-3779-4cf7-8a98-2e51939e7639" (UID: "63eddb85-3779-4cf7-8a98-2e51939e7639"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:05:27.960985 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:27.960944 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63eddb85-3779-4cf7-8a98-2e51939e7639-kube-api-access-wpsld" pod "63eddb85-3779-4cf7-8a98-2e51939e7639" (UID: "63eddb85-3779-4cf7-8a98-2e51939e7639"). InnerVolumeSpecName "kube-api-access-wpsld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:27.963756 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:27.963726 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-xks-ca" pod "63eddb85-3779-4cf7-8a98-2e51939e7639" (UID: "63eddb85-3779-4cf7-8a98-2e51939e7639"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:05:28.055635 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:28.055512 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:28.055635 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:28.055554 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpsld\" (UniqueName: \"kubernetes.io/projected/63eddb85-3779-4cf7-8a98-2e51939e7639-kube-api-access-wpsld\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:28.055635 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:28.055570 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63eddb85-3779-4cf7-8a98-2e51939e7639-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:28.682806 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:28.682781 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" Oct 31 00:05:28.682981 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:28.682776 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z" event={"ID":"63eddb85-3779-4cf7-8a98-2e51939e7639","Type":"ContainerDied","Data":"a202a474d452e131e6c99fa42bc96d98c753898f1a034174f921e236a23a1c33"} Oct 31 00:05:28.682981 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:28.682926 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a202a474d452e131e6c99fa42bc96d98c753898f1a034174f921e236a23a1c33" Oct 31 00:05:28.774510 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:05:28.774477 7570 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63eddb85_3779_4cf7_8a98_2e51939e7639.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:05:29.030296 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:29.030194 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn"] Oct 31 00:05:29.054226 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:29.054192 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h"] Oct 31 00:05:29.054485 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:29.054448 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d0f229-4kzgn"] Oct 31 00:05:29.054636 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:29.054622 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeee9160-4wv2h"] Oct 31 00:05:29.217600 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:29.217563 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:05:29.217881 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:29.217850 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:05:29.235572 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:29.235541 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:05:29.235794 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:29.235769 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:05:29.483750 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:05:29.483719 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:05:30.608680 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:30.608647 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4f82dd-efe7-4cfc-a4b3-507773e99758" path="/var/lib/kubelet/pods/8b4f82dd-efe7-4cfc-a4b3-507773e99758/volumes" Oct 31 00:05:30.609076 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:30.609063 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94b6a9a-db8d-4e81-affe-c11f0e402d6b" path="/var/lib/kubelet/pods/e94b6a9a-db8d-4e81-affe-c11f0e402d6b/volumes" Oct 31 00:05:43.827756 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:43.827726 7570 scope.go:122] "RemoveContainer" containerID="ea56ec98a7b7183ec6533b531a5a5c1a2a735c37901b53015239cd43e8e4790b" Oct 31 00:05:43.837522 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:43.837494 7570 scope.go:122] "RemoveContainer" containerID="a6ef2d1f1857336abefb165a86f274daf6e7a5f5213d48285f752db7f46e54a7" Oct 31 00:05:43.846228 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:43.846211 7570 scope.go:122] "RemoveContainer" containerID="1b2b11e55ea179c0a86d92b0ea3376b0e745720c9f4ce8971e68e8f35f75dfab" Oct 31 00:05:43.854792 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:43.854776 7570 scope.go:122] "RemoveContainer" containerID="e4860eb07968b5d32b9ae757e4cedbd8eba31fdeaaf9615e65b7f3ce75b3253e" Oct 31 00:05:43.862925 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:43.862909 7570 scope.go:122] "RemoveContainer" containerID="2cbcced42f15a0d21b9f955bccb9218fbf20b65e04929c9390f0728f19804dd0" Oct 31 00:05:51.122043 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.122011 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv"] Oct 31 00:05:51.122427 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.122418 7570 cpu_manager.go:414] "containerMap: removing container" podUID="63eddb85-3779-4cf7-8a98-2e51939e7639" containerName="revoke-keys" Oct 31 00:05:51.122486 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.122429 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63eddb85-3779-4cf7-8a98-2e51939e7639" containerName="revoke-keys" Oct 31 00:05:51.122549 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.122537 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="63eddb85-3779-4cf7-8a98-2e51939e7639" containerName="revoke-keys" Oct 31 00:05:51.125667 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.125653 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:51.132216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.132192 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:05:51.132406 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.132385 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:05:51.132406 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.132293 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:05:51.133292 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.133267 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:05:51.134784 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.134764 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv"] Oct 31 00:05:51.212627 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.212594 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-txkxv\" (UID: \"252ce7ac-016d-4803-af32-b0812df8d361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:51.212821 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.212726 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-txkxv\" (UID: \"252ce7ac-016d-4803-af32-b0812df8d361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:51.212821 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.212780 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-292t4\" (UniqueName: \"kubernetes.io/projected/252ce7ac-016d-4803-af32-b0812df8d361-kube-api-access-292t4\") pod \"maas-api-revoke-sub-046e1343-txkxv\" (UID: \"252ce7ac-016d-4803-af32-b0812df8d361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:51.314084 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.314053 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-txkxv\" (UID: \"252ce7ac-016d-4803-af32-b0812df8d361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:51.314257 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.314104 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-txkxv\" (UID: \"252ce7ac-016d-4803-af32-b0812df8d361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:51.314257 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.314188 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-292t4\" (UniqueName: \"kubernetes.io/projected/252ce7ac-016d-4803-af32-b0812df8d361-kube-api-access-292t4\") pod \"maas-api-revoke-sub-046e1343-txkxv\" (UID: \"252ce7ac-016d-4803-af32-b0812df8d361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:51.314843 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.314784 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-txkxv\" (UID: \"252ce7ac-016d-4803-af32-b0812df8d361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:51.316606 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.316585 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-txkxv\" (UID: \"252ce7ac-016d-4803-af32-b0812df8d361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:51.327412 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.327394 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-292t4\" (UniqueName: \"kubernetes.io/projected/252ce7ac-016d-4803-af32-b0812df8d361-kube-api-access-292t4\") pod \"maas-api-revoke-sub-046e1343-txkxv\" (UID: \"252ce7ac-016d-4803-af32-b0812df8d361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:51.441089 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.441028 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:51.719050 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.719021 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv"] Oct 31 00:05:51.720812 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:05:51.720785 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252ce7ac_016d_4803_af32_b0812df8d361.slice/crio-ca0641560c99c8489b4d70d5c98a812de77259b32bf132ef555d2377b0a7a1ab WatchSource:0}: Error finding container ca0641560c99c8489b4d70d5c98a812de77259b32bf132ef555d2377b0a7a1ab: Status 404 returned error can't find the container with id ca0641560c99c8489b4d70d5c98a812de77259b32bf132ef555d2377b0a7a1ab Oct 31 00:05:51.832068 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:51.832040 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" event={"ID":"252ce7ac-016d-4803-af32-b0812df8d361","Type":"ContainerStarted","Data":"ca0641560c99c8489b4d70d5c98a812de77259b32bf132ef555d2377b0a7a1ab"} Oct 31 00:05:52.839166 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:52.839133 7570 generic.go:356] "Generic (PLEG): container finished" podID="252ce7ac-016d-4803-af32-b0812df8d361" containerID="40dcebdf055fcfd90b7f6e9a4944dc8f7f3ce4daef6002b149047d71736da467" exitCode=0 Oct 31 00:05:52.839610 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:52.839221 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" event={"ID":"252ce7ac-016d-4803-af32-b0812df8d361","Type":"ContainerDied","Data":"40dcebdf055fcfd90b7f6e9a4944dc8f7f3ce4daef6002b149047d71736da467"} Oct 31 00:05:53.985801 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:53.985780 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:54.143388 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:54.143314 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-openshift-ca\") pod \"252ce7ac-016d-4803-af32-b0812df8d361\" (UID: \"252ce7ac-016d-4803-af32-b0812df8d361\") " Oct 31 00:05:54.143599 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:54.143425 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-xks-ca\") pod \"252ce7ac-016d-4803-af32-b0812df8d361\" (UID: \"252ce7ac-016d-4803-af32-b0812df8d361\") " Oct 31 00:05:54.143599 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:54.143506 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252ce7ac-016d-4803-af32-b0812df8d361-kube-api-access-292t4\" (UniqueName: \"kubernetes.io/projected/252ce7ac-016d-4803-af32-b0812df8d361-kube-api-access-292t4\") pod \"252ce7ac-016d-4803-af32-b0812df8d361\" (UID: \"252ce7ac-016d-4803-af32-b0812df8d361\") " Oct 31 00:05:54.143724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:54.143672 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-openshift-ca" pod "252ce7ac-016d-4803-af32-b0812df8d361" (UID: "252ce7ac-016d-4803-af32-b0812df8d361"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:05:54.143796 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:54.143779 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:54.145821 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:54.145798 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-xks-ca" pod "252ce7ac-016d-4803-af32-b0812df8d361" (UID: "252ce7ac-016d-4803-af32-b0812df8d361"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:05:54.145908 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:54.145826 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252ce7ac-016d-4803-af32-b0812df8d361-kube-api-access-292t4" pod "252ce7ac-016d-4803-af32-b0812df8d361" (UID: "252ce7ac-016d-4803-af32-b0812df8d361"). InnerVolumeSpecName "kube-api-access-292t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:54.244661 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:54.244633 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252ce7ac-016d-4803-af32-b0812df8d361-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:54.244661 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:54.244659 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-292t4\" (UniqueName: \"kubernetes.io/projected/252ce7ac-016d-4803-af32-b0812df8d361-kube-api-access-292t4\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:54.852489 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:54.852436 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" Oct 31 00:05:54.852654 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:54.852489 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv" event={"ID":"252ce7ac-016d-4803-af32-b0812df8d361","Type":"ContainerDied","Data":"ca0641560c99c8489b4d70d5c98a812de77259b32bf132ef555d2377b0a7a1ab"} Oct 31 00:05:54.852654 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:05:54.852560 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca0641560c99c8489b4d70d5c98a812de77259b32bf132ef555d2377b0a7a1ab" Oct 31 00:05:59.411758 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:05:59.411722 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:06:01.358968 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.358936 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz"] Oct 31 00:06:01.359414 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.359400 7570 cpu_manager.go:414] "containerMap: removing container" podUID="252ce7ac-016d-4803-af32-b0812df8d361" containerName="revoke-keys" Oct 31 00:06:01.359482 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.359416 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252ce7ac-016d-4803-af32-b0812df8d361" containerName="revoke-keys" Oct 31 00:06:01.359551 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.359541 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="252ce7ac-016d-4803-af32-b0812df8d361" containerName="revoke-keys" Oct 31 00:06:01.363542 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.363524 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:01.370697 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.369041 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:06:01.370697 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.369169 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:06:01.370697 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.369216 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:06:01.370697 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.369272 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:06:01.372636 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.372612 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz"] Oct 31 00:06:01.515232 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.515183 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8n8c\" (UniqueName: \"kubernetes.io/projected/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-kube-api-access-q8n8c\") pod \"maas-api-revoke-sub-d3fba056-x5hkz\" (UID: \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:01.515396 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.515248 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-x5hkz\" (UID: \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:01.515451 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.515432 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-x5hkz\" (UID: \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:01.616872 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.616789 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8n8c\" (UniqueName: \"kubernetes.io/projected/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-kube-api-access-q8n8c\") pod \"maas-api-revoke-sub-d3fba056-x5hkz\" (UID: \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:01.616872 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.616844 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-x5hkz\" (UID: \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:01.617074 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.616972 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-x5hkz\" (UID: \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:01.617547 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.617520 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-x5hkz\" (UID: \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:01.619531 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.619512 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-x5hkz\" (UID: \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:01.629164 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.629142 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8n8c\" (UniqueName: \"kubernetes.io/projected/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-kube-api-access-q8n8c\") pod \"maas-api-revoke-sub-d3fba056-x5hkz\" (UID: \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:01.684756 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.684735 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:01.962658 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:01.962632 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz"] Oct 31 00:06:01.964233 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:06:01.964192 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8065de8_6c7b_4b7b_a044_0cebb0f930a7.slice/crio-9bf0690b6ab166e99bfaa39542aeebd6a26ff00c2df0b7d215a8b90e42f35a5d WatchSource:0}: Error finding container 9bf0690b6ab166e99bfaa39542aeebd6a26ff00c2df0b7d215a8b90e42f35a5d: Status 404 returned error can't find the container with id 9bf0690b6ab166e99bfaa39542aeebd6a26ff00c2df0b7d215a8b90e42f35a5d Oct 31 00:06:02.909236 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:02.909194 7570 generic.go:356] "Generic (PLEG): container finished" podID="f8065de8-6c7b-4b7b-a044-0cebb0f930a7" containerID="4015fab92b382ade63c02ce21e5ee6d9c1f855a08a44ca8ed5ce6261e51804c9" exitCode=0 Oct 31 00:06:02.909678 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:02.909287 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" event={"ID":"f8065de8-6c7b-4b7b-a044-0cebb0f930a7","Type":"ContainerDied","Data":"4015fab92b382ade63c02ce21e5ee6d9c1f855a08a44ca8ed5ce6261e51804c9"} Oct 31 00:06:02.909678 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:02.909352 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" event={"ID":"f8065de8-6c7b-4b7b-a044-0cebb0f930a7","Type":"ContainerStarted","Data":"9bf0690b6ab166e99bfaa39542aeebd6a26ff00c2df0b7d215a8b90e42f35a5d"} Oct 31 00:06:04.314180 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.313807 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:04.376612 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.376576 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-kube-api-access-q8n8c\" (UniqueName: \"kubernetes.io/projected/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-kube-api-access-q8n8c\") pod \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\" (UID: \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\") " Oct 31 00:06:04.376810 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.376749 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-xks-ca\") pod \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\" (UID: \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\") " Oct 31 00:06:04.376810 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.376779 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-openshift-ca\") pod \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\" (UID: \"f8065de8-6c7b-4b7b-a044-0cebb0f930a7\") " Oct 31 00:06:04.377501 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.377447 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-openshift-ca" pod "f8065de8-6c7b-4b7b-a044-0cebb0f930a7" (UID: "f8065de8-6c7b-4b7b-a044-0cebb0f930a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:04.379014 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.378983 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-kube-api-access-q8n8c" pod "f8065de8-6c7b-4b7b-a044-0cebb0f930a7" (UID: "f8065de8-6c7b-4b7b-a044-0cebb0f930a7"). InnerVolumeSpecName "kube-api-access-q8n8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:04.379589 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.379555 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-xks-ca" pod "f8065de8-6c7b-4b7b-a044-0cebb0f930a7" (UID: "f8065de8-6c7b-4b7b-a044-0cebb0f930a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:04.478088 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.478049 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:04.478088 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.478082 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:04.478280 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.478097 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8n8c\" (UniqueName: \"kubernetes.io/projected/f8065de8-6c7b-4b7b-a044-0cebb0f930a7-kube-api-access-q8n8c\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:04.957702 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.957612 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" Oct 31 00:06:04.957876 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.957605 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz" event={"ID":"f8065de8-6c7b-4b7b-a044-0cebb0f930a7","Type":"ContainerDied","Data":"9bf0690b6ab166e99bfaa39542aeebd6a26ff00c2df0b7d215a8b90e42f35a5d"} Oct 31 00:06:04.957876 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:04.957759 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bf0690b6ab166e99bfaa39542aeebd6a26ff00c2df0b7d215a8b90e42f35a5d" Oct 31 00:06:22.430510 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.430478 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6"] Oct 31 00:06:22.431024 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.430980 7570 cpu_manager.go:414] "containerMap: removing container" podUID="f8065de8-6c7b-4b7b-a044-0cebb0f930a7" containerName="revoke-keys" Oct 31 00:06:22.431024 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.430993 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8065de8-6c7b-4b7b-a044-0cebb0f930a7" containerName="revoke-keys" Oct 31 00:06:22.431095 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.431081 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8065de8-6c7b-4b7b-a044-0cebb0f930a7" containerName="revoke-keys" Oct 31 00:06:22.434265 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.434247 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:22.439366 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.439341 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:06:22.440747 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.440725 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:06:22.441187 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.441164 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:06:22.441306 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.441256 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:06:22.445190 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.445161 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6"] Oct 31 00:06:22.614573 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.614542 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9lsl\" (UniqueName: \"kubernetes.io/projected/76dc2375-44d4-42ab-b717-7bafbce8eb97-kube-api-access-d9lsl\") pod \"maas-api-revoke-sub-ae594979-sgtq6\" (UID: \"76dc2375-44d4-42ab-b717-7bafbce8eb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:22.614757 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.614715 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-sgtq6\" (UID: \"76dc2375-44d4-42ab-b717-7bafbce8eb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:22.614819 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.614784 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-sgtq6\" (UID: \"76dc2375-44d4-42ab-b717-7bafbce8eb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:22.715357 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.715279 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-sgtq6\" (UID: \"76dc2375-44d4-42ab-b717-7bafbce8eb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:22.715357 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.715345 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-sgtq6\" (UID: \"76dc2375-44d4-42ab-b717-7bafbce8eb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:22.715545 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.715382 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9lsl\" (UniqueName: \"kubernetes.io/projected/76dc2375-44d4-42ab-b717-7bafbce8eb97-kube-api-access-d9lsl\") pod \"maas-api-revoke-sub-ae594979-sgtq6\" (UID: \"76dc2375-44d4-42ab-b717-7bafbce8eb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:22.716106 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.716086 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-sgtq6\" (UID: \"76dc2375-44d4-42ab-b717-7bafbce8eb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:22.717861 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.717841 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-sgtq6\" (UID: \"76dc2375-44d4-42ab-b717-7bafbce8eb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:22.727681 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.727662 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9lsl\" (UniqueName: \"kubernetes.io/projected/76dc2375-44d4-42ab-b717-7bafbce8eb97-kube-api-access-d9lsl\") pod \"maas-api-revoke-sub-ae594979-sgtq6\" (UID: \"76dc2375-44d4-42ab-b717-7bafbce8eb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:22.749543 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:22.749521 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:23.027210 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:23.027183 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6"] Oct 31 00:06:23.028508 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:06:23.028453 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dc2375_44d4_42ab_b717_7bafbce8eb97.slice/crio-4a5632c92edfe403abb51ce83bf0678de2e438df0aff9e9c67760a584a813bb7 WatchSource:0}: Error finding container 4a5632c92edfe403abb51ce83bf0678de2e438df0aff9e9c67760a584a813bb7: Status 404 returned error can't find the container with id 4a5632c92edfe403abb51ce83bf0678de2e438df0aff9e9c67760a584a813bb7 Oct 31 00:06:23.126403 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:23.126370 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" event={"ID":"76dc2375-44d4-42ab-b717-7bafbce8eb97","Type":"ContainerStarted","Data":"4a5632c92edfe403abb51ce83bf0678de2e438df0aff9e9c67760a584a813bb7"} Oct 31 00:06:24.133890 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:24.133852 7570 generic.go:356] "Generic (PLEG): container finished" podID="76dc2375-44d4-42ab-b717-7bafbce8eb97" containerID="64eb8eb10a549b03fe197616e24eae1977304e486a87c240a47a91cb341a8684" exitCode=0 Oct 31 00:06:24.134297 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:24.133898 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" event={"ID":"76dc2375-44d4-42ab-b717-7bafbce8eb97","Type":"ContainerDied","Data":"64eb8eb10a549b03fe197616e24eae1977304e486a87c240a47a91cb341a8684"} Oct 31 00:06:25.274077 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:25.274055 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:25.339585 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:25.339559 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-openshift-ca\") pod \"76dc2375-44d4-42ab-b717-7bafbce8eb97\" (UID: \"76dc2375-44d4-42ab-b717-7bafbce8eb97\") " Oct 31 00:06:25.339746 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:25.339599 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-xks-ca\") pod \"76dc2375-44d4-42ab-b717-7bafbce8eb97\" (UID: \"76dc2375-44d4-42ab-b717-7bafbce8eb97\") " Oct 31 00:06:25.339746 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:25.339649 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76dc2375-44d4-42ab-b717-7bafbce8eb97-kube-api-access-d9lsl\" (UniqueName: \"kubernetes.io/projected/76dc2375-44d4-42ab-b717-7bafbce8eb97-kube-api-access-d9lsl\") pod \"76dc2375-44d4-42ab-b717-7bafbce8eb97\" (UID: \"76dc2375-44d4-42ab-b717-7bafbce8eb97\") " Oct 31 00:06:25.339974 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:25.339951 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-openshift-ca" pod "76dc2375-44d4-42ab-b717-7bafbce8eb97" (UID: "76dc2375-44d4-42ab-b717-7bafbce8eb97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:25.341899 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:25.341871 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dc2375-44d4-42ab-b717-7bafbce8eb97-kube-api-access-d9lsl" pod "76dc2375-44d4-42ab-b717-7bafbce8eb97" (UID: "76dc2375-44d4-42ab-b717-7bafbce8eb97"). InnerVolumeSpecName "kube-api-access-d9lsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:25.341899 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:25.341879 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-xks-ca" pod "76dc2375-44d4-42ab-b717-7bafbce8eb97" (UID: "76dc2375-44d4-42ab-b717-7bafbce8eb97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:25.440746 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:25.440681 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:25.440746 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:25.440709 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76dc2375-44d4-42ab-b717-7bafbce8eb97-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:25.440746 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:25.440719 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9lsl\" (UniqueName: \"kubernetes.io/projected/76dc2375-44d4-42ab-b717-7bafbce8eb97-kube-api-access-d9lsl\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:26.149037 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:26.149005 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" Oct 31 00:06:26.149227 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:26.149041 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6" event={"ID":"76dc2375-44d4-42ab-b717-7bafbce8eb97","Type":"ContainerDied","Data":"4a5632c92edfe403abb51ce83bf0678de2e438df0aff9e9c67760a584a813bb7"} Oct 31 00:06:26.149227 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:26.149087 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a5632c92edfe403abb51ce83bf0678de2e438df0aff9e9c67760a584a813bb7" Oct 31 00:06:29.411485 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:06:29.411432 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:06:37.036563 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:37.036523 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4"] Oct 31 00:06:37.052492 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:37.048526 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6"] Oct 31 00:06:37.071542 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:37.071509 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh"] Oct 31 00:06:37.071777 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:37.071762 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7af98d5a-m66c4"] Oct 31 00:06:37.073693 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:37.073547 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fbcbc45-htxw6"] Oct 31 00:06:37.077038 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:37.077011 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64d94e2-wl5kh"] Oct 31 00:06:38.601359 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:38.601327 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab1e878-bca9-41c4-81e6-0001487f5ff1" path="/var/lib/kubelet/pods/1ab1e878-bca9-41c4-81e6-0001487f5ff1/volumes" Oct 31 00:06:38.601766 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:38.601744 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28216477-7eed-4e96-a7e7-262ab1d60bb9" path="/var/lib/kubelet/pods/28216477-7eed-4e96-a7e7-262ab1d60bb9/volumes" Oct 31 00:06:38.602104 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:38.602090 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47465535-f16b-4864-a764-71f07c0d273d" path="/var/lib/kubelet/pods/47465535-f16b-4864-a764-71f07c0d273d/volumes" Oct 31 00:06:40.490719 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.490681 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7"] Oct 31 00:06:40.491240 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.491225 7570 cpu_manager.go:414] "containerMap: removing container" podUID="76dc2375-44d4-42ab-b717-7bafbce8eb97" containerName="revoke-keys" Oct 31 00:06:40.491290 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.491245 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76dc2375-44d4-42ab-b717-7bafbce8eb97" containerName="revoke-keys" Oct 31 00:06:40.491351 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.491341 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="76dc2375-44d4-42ab-b717-7bafbce8eb97" containerName="revoke-keys" Oct 31 00:06:40.495976 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.495961 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:40.500838 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.500814 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:06:40.500999 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.500978 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:06:40.503910 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.503886 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7"] Oct 31 00:06:40.545164 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.545133 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:06:40.545292 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.545173 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:06:40.604553 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.604528 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8s7b7\" (UID: \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:40.604681 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.604599 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8s7b7\" (UID: \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:40.604681 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.604655 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg72g\" (UniqueName: \"kubernetes.io/projected/714f825a-d72f-4cc6-b14a-4b71590b2ce3-kube-api-access-jg72g\") pod \"maas-api-revoke-sub-a4aef7d7-8s7b7\" (UID: \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:40.705517 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.705485 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8s7b7\" (UID: \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:40.705673 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.705543 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8s7b7\" (UID: \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:40.705673 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.705567 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg72g\" (UniqueName: \"kubernetes.io/projected/714f825a-d72f-4cc6-b14a-4b71590b2ce3-kube-api-access-jg72g\") pod \"maas-api-revoke-sub-a4aef7d7-8s7b7\" (UID: \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:40.706173 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.706150 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8s7b7\" (UID: \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:40.708058 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.708041 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8s7b7\" (UID: \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:40.717971 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.717951 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg72g\" (UniqueName: \"kubernetes.io/projected/714f825a-d72f-4cc6-b14a-4b71590b2ce3-kube-api-access-jg72g\") pod \"maas-api-revoke-sub-a4aef7d7-8s7b7\" (UID: \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:40.810455 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:40.810393 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:41.091250 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:41.091223 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7"] Oct 31 00:06:41.092614 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:06:41.092589 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714f825a_d72f_4cc6_b14a_4b71590b2ce3.slice/crio-c7ed155081c64796c2e3f982dc8cbab0e65e61a185fb50fe7c886ebb3e415349 WatchSource:0}: Error finding container c7ed155081c64796c2e3f982dc8cbab0e65e61a185fb50fe7c886ebb3e415349: Status 404 returned error can't find the container with id c7ed155081c64796c2e3f982dc8cbab0e65e61a185fb50fe7c886ebb3e415349 Oct 31 00:06:41.286760 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:41.286727 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" event={"ID":"714f825a-d72f-4cc6-b14a-4b71590b2ce3","Type":"ContainerStarted","Data":"c7ed155081c64796c2e3f982dc8cbab0e65e61a185fb50fe7c886ebb3e415349"} Oct 31 00:06:42.295389 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:42.295305 7570 generic.go:356] "Generic (PLEG): container finished" podID="714f825a-d72f-4cc6-b14a-4b71590b2ce3" containerID="e2f966c9f488e25d24c13f6a147bd7b336a88d40f41259de8a0fff6649e3d363" exitCode=0 Oct 31 00:06:42.295777 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:42.295394 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" event={"ID":"714f825a-d72f-4cc6-b14a-4b71590b2ce3","Type":"ContainerDied","Data":"e2f966c9f488e25d24c13f6a147bd7b336a88d40f41259de8a0fff6649e3d363"} Oct 31 00:06:43.446377 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.446357 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:43.533812 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.533761 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714f825a-d72f-4cc6-b14a-4b71590b2ce3-kube-api-access-jg72g\" (UniqueName: \"kubernetes.io/projected/714f825a-d72f-4cc6-b14a-4b71590b2ce3-kube-api-access-jg72g\") pod \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\" (UID: \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\") " Oct 31 00:06:43.533984 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.533850 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-xks-ca\") pod \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\" (UID: \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\") " Oct 31 00:06:43.533984 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.533922 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-openshift-ca\") pod \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\" (UID: \"714f825a-d72f-4cc6-b14a-4b71590b2ce3\") " Oct 31 00:06:43.534290 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.534255 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-openshift-ca" pod "714f825a-d72f-4cc6-b14a-4b71590b2ce3" (UID: "714f825a-d72f-4cc6-b14a-4b71590b2ce3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:43.536141 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.536105 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-xks-ca" pod "714f825a-d72f-4cc6-b14a-4b71590b2ce3" (UID: "714f825a-d72f-4cc6-b14a-4b71590b2ce3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:43.536141 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.536138 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714f825a-d72f-4cc6-b14a-4b71590b2ce3-kube-api-access-jg72g" pod "714f825a-d72f-4cc6-b14a-4b71590b2ce3" (UID: "714f825a-d72f-4cc6-b14a-4b71590b2ce3"). InnerVolumeSpecName "kube-api-access-jg72g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:43.635531 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.635494 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:43.635531 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.635529 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg72g\" (UniqueName: \"kubernetes.io/projected/714f825a-d72f-4cc6-b14a-4b71590b2ce3-kube-api-access-jg72g\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:43.635531 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.635538 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714f825a-d72f-4cc6-b14a-4b71590b2ce3-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:43.916612 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.916531 7570 scope.go:122] "RemoveContainer" containerID="55411c1657d957f452c90681a8c1bc2120c505043443a28cc6be687ffe62d5a9" Oct 31 00:06:43.926642 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.926621 7570 scope.go:122] "RemoveContainer" containerID="46b44bfd009290a9a959d8a63a0e60c9bd5890dd23edf0a8ca5204157b481c63" Oct 31 00:06:43.935323 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.935303 7570 scope.go:122] "RemoveContainer" containerID="4906c75837cc7844108faedf93543b16de0b5b30f08d771d3188a7bbfb76b3c4" Oct 31 00:06:43.943824 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.943805 7570 scope.go:122] "RemoveContainer" containerID="4f9993d222c0b9881fb05b9ec50d101dd246b4fbe13fc95f97d48f2bea530a7f" Oct 31 00:06:43.952557 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.952533 7570 scope.go:122] "RemoveContainer" containerID="6030f48318c3894c897c8c86533e531df88e1cc296d9190323ca6e8979d74118" Oct 31 00:06:43.961069 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.961047 7570 scope.go:122] "RemoveContainer" containerID="ff4818a396ddb4cb0ae97902881cf4ad80caa0c89f34ade12ce4ea534ba03c54" Oct 31 00:06:43.970190 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:43.970172 7570 scope.go:122] "RemoveContainer" containerID="bb710b18e03c8521ecbad0d2d43f7fd329c53203d225ed9b478bc4a747be60b6" Oct 31 00:06:44.310747 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:44.310665 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" event={"ID":"714f825a-d72f-4cc6-b14a-4b71590b2ce3","Type":"ContainerDied","Data":"c7ed155081c64796c2e3f982dc8cbab0e65e61a185fb50fe7c886ebb3e415349"} Oct 31 00:06:44.310747 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:44.310694 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7" Oct 31 00:06:44.310747 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:44.310715 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7ed155081c64796c2e3f982dc8cbab0e65e61a185fb50fe7c886ebb3e415349" Oct 31 00:06:59.325027 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.324991 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq"] Oct 31 00:06:59.325672 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.325447 7570 cpu_manager.go:414] "containerMap: removing container" podUID="714f825a-d72f-4cc6-b14a-4b71590b2ce3" containerName="revoke-keys" Oct 31 00:06:59.325672 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.325476 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714f825a-d72f-4cc6-b14a-4b71590b2ce3" containerName="revoke-keys" Oct 31 00:06:59.325672 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.325550 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="714f825a-d72f-4cc6-b14a-4b71590b2ce3" containerName="revoke-keys" Oct 31 00:06:59.328834 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.328816 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:06:59.333181 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.333158 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:06:59.334348 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.334307 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:06:59.334498 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.334345 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:06:59.334639 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.334616 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:06:59.337383 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.337361 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq"] Oct 31 00:06:59.398818 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.398788 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg666\" (UniqueName: \"kubernetes.io/projected/e3b4f6b9-edb6-4f70-b108-087dec2e2093-kube-api-access-wg666\") pod \"maas-api-revoke-sub-79805ca9-2tqwq\" (UID: \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:06:59.399014 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.398848 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2tqwq\" (UID: \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:06:59.399077 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.399030 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2tqwq\" (UID: \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:06:59.415875 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:06:59.415841 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:06:59.500666 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.500637 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg666\" (UniqueName: \"kubernetes.io/projected/e3b4f6b9-edb6-4f70-b108-087dec2e2093-kube-api-access-wg666\") pod \"maas-api-revoke-sub-79805ca9-2tqwq\" (UID: \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:06:59.500820 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.500724 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2tqwq\" (UID: \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:06:59.500820 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.500788 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2tqwq\" (UID: \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:06:59.501384 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.501358 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2tqwq\" (UID: \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:06:59.503287 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.503264 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2tqwq\" (UID: \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:06:59.512957 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.512930 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg666\" (UniqueName: \"kubernetes.io/projected/e3b4f6b9-edb6-4f70-b108-087dec2e2093-kube-api-access-wg666\") pod \"maas-api-revoke-sub-79805ca9-2tqwq\" (UID: \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:06:59.645782 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.645734 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:06:59.923499 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:06:59.923444 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq"] Oct 31 00:06:59.925603 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:06:59.925564 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b4f6b9_edb6_4f70_b108_087dec2e2093.slice/crio-ad3eae01ebe41de5f40ff0867dd3a39e4e7d2dd0285daeca17e3065d9bab52b3 WatchSource:0}: Error finding container ad3eae01ebe41de5f40ff0867dd3a39e4e7d2dd0285daeca17e3065d9bab52b3: Status 404 returned error can't find the container with id ad3eae01ebe41de5f40ff0867dd3a39e4e7d2dd0285daeca17e3065d9bab52b3 Oct 31 00:07:00.411012 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:00.410974 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" event={"ID":"e3b4f6b9-edb6-4f70-b108-087dec2e2093","Type":"ContainerStarted","Data":"ea82d2844320ed986eb1fc6c01320fe5fc742837f956232e9702b145ad34ab3a"} Oct 31 00:07:00.411411 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:00.411030 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" event={"ID":"e3b4f6b9-edb6-4f70-b108-087dec2e2093","Type":"ContainerStarted","Data":"ad3eae01ebe41de5f40ff0867dd3a39e4e7d2dd0285daeca17e3065d9bab52b3"} Oct 31 00:07:00.429636 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:00.429590 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" podStartSLOduration=1.429575794 podStartE2EDuration="1.429575794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:00.428883885 +0000 UTC m=+2831.947884976" watchObservedRunningTime="2026-10-31 00:07:00.429575794 +0000 UTC m=+2831.948576884" Oct 31 00:07:01.422139 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:01.422106 7570 generic.go:356] "Generic (PLEG): container finished" podID="e3b4f6b9-edb6-4f70-b108-087dec2e2093" containerID="ea82d2844320ed986eb1fc6c01320fe5fc742837f956232e9702b145ad34ab3a" exitCode=0 Oct 31 00:07:01.422598 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:01.422155 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" event={"ID":"e3b4f6b9-edb6-4f70-b108-087dec2e2093","Type":"ContainerDied","Data":"ea82d2844320ed986eb1fc6c01320fe5fc742837f956232e9702b145ad34ab3a"} Oct 31 00:07:02.569126 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:02.569104 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:07:02.634539 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:02.634514 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-openshift-ca\") pod \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\" (UID: \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\") " Oct 31 00:07:02.634681 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:02.634596 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-xks-ca\") pod \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\" (UID: \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\") " Oct 31 00:07:02.634681 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:02.634632 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b4f6b9-edb6-4f70-b108-087dec2e2093-kube-api-access-wg666\" (UniqueName: \"kubernetes.io/projected/e3b4f6b9-edb6-4f70-b108-087dec2e2093-kube-api-access-wg666\") pod \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\" (UID: \"e3b4f6b9-edb6-4f70-b108-087dec2e2093\") " Oct 31 00:07:02.634855 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:02.634831 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-openshift-ca" pod "e3b4f6b9-edb6-4f70-b108-087dec2e2093" (UID: "e3b4f6b9-edb6-4f70-b108-087dec2e2093"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:02.636821 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:02.636800 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-xks-ca" pod "e3b4f6b9-edb6-4f70-b108-087dec2e2093" (UID: "e3b4f6b9-edb6-4f70-b108-087dec2e2093"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:02.636909 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:02.636860 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b4f6b9-edb6-4f70-b108-087dec2e2093-kube-api-access-wg666" pod "e3b4f6b9-edb6-4f70-b108-087dec2e2093" (UID: "e3b4f6b9-edb6-4f70-b108-087dec2e2093"). InnerVolumeSpecName "kube-api-access-wg666". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:02.735796 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:02.735730 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg666\" (UniqueName: \"kubernetes.io/projected/e3b4f6b9-edb6-4f70-b108-087dec2e2093-kube-api-access-wg666\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:02.735796 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:02.735761 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:02.735796 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:02.735778 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b4f6b9-edb6-4f70-b108-087dec2e2093-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:03.436199 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:03.436166 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" Oct 31 00:07:03.436365 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:03.436198 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq" event={"ID":"e3b4f6b9-edb6-4f70-b108-087dec2e2093","Type":"ContainerDied","Data":"ad3eae01ebe41de5f40ff0867dd3a39e4e7d2dd0285daeca17e3065d9bab52b3"} Oct 31 00:07:03.436365 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:03.436250 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad3eae01ebe41de5f40ff0867dd3a39e4e7d2dd0285daeca17e3065d9bab52b3" Oct 31 00:07:07.023030 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:07.022993 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v"] Oct 31 00:07:07.025654 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:07.025627 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e04d396-bkt7v"] Oct 31 00:07:08.600883 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:08.600856 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9ded63-bf41-43a1-8b50-33fadb1c57fe" path="/var/lib/kubelet/pods/1d9ded63-bf41-43a1-8b50-33fadb1c57fe/volumes" Oct 31 00:07:14.235487 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:14.234560 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-7rp6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:07:14.235487 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:14.234636 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-7rp6c" podUID="e7406dbc-b7ad-4b0c-ac37-c85b302bf76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:07:17.124963 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.124933 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn"] Oct 31 00:07:17.125367 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.125354 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e3b4f6b9-edb6-4f70-b108-087dec2e2093" containerName="revoke-keys" Oct 31 00:07:17.125409 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.125368 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b4f6b9-edb6-4f70-b108-087dec2e2093" containerName="revoke-keys" Oct 31 00:07:17.125483 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.125474 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b4f6b9-edb6-4f70-b108-087dec2e2093" containerName="revoke-keys" Oct 31 00:07:17.128657 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.128641 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:17.134377 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.133659 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:07:17.134377 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.133709 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:07:17.134377 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.133798 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:07:17.135093 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.135072 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:07:17.138715 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.138692 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn"] Oct 31 00:07:17.289143 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.289111 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ckpv\" (UniqueName: \"kubernetes.io/projected/60ef3464-bc1e-4efa-b829-7264ef415ab7-kube-api-access-7ckpv\") pod \"maas-api-revoke-sub-0b7e8edb-6v7sn\" (UID: \"60ef3464-bc1e-4efa-b829-7264ef415ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:17.289310 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.289291 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6v7sn\" (UID: \"60ef3464-bc1e-4efa-b829-7264ef415ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:17.289369 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.289323 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6v7sn\" (UID: \"60ef3464-bc1e-4efa-b829-7264ef415ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:17.390716 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.390641 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6v7sn\" (UID: \"60ef3464-bc1e-4efa-b829-7264ef415ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:17.390716 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.390673 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6v7sn\" (UID: \"60ef3464-bc1e-4efa-b829-7264ef415ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:17.390716 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.390699 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ckpv\" (UniqueName: \"kubernetes.io/projected/60ef3464-bc1e-4efa-b829-7264ef415ab7-kube-api-access-7ckpv\") pod \"maas-api-revoke-sub-0b7e8edb-6v7sn\" (UID: \"60ef3464-bc1e-4efa-b829-7264ef415ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:17.391273 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.391251 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6v7sn\" (UID: \"60ef3464-bc1e-4efa-b829-7264ef415ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:17.393242 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.393217 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6v7sn\" (UID: \"60ef3464-bc1e-4efa-b829-7264ef415ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:17.403570 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.403545 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ckpv\" (UniqueName: \"kubernetes.io/projected/60ef3464-bc1e-4efa-b829-7264ef415ab7-kube-api-access-7ckpv\") pod \"maas-api-revoke-sub-0b7e8edb-6v7sn\" (UID: \"60ef3464-bc1e-4efa-b829-7264ef415ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:17.444367 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.444342 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:17.722051 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:17.722022 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn"] Oct 31 00:07:17.723752 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:07:17.723726 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ef3464_bc1e_4efa_b829_7264ef415ab7.slice/crio-da546cfeed19b143043b2a5348c996dcafcd9af328fddf7b7101e5d95bc06891 WatchSource:0}: Error finding container da546cfeed19b143043b2a5348c996dcafcd9af328fddf7b7101e5d95bc06891: Status 404 returned error can't find the container with id da546cfeed19b143043b2a5348c996dcafcd9af328fddf7b7101e5d95bc06891 Oct 31 00:07:18.022924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:18.022901 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x"] Oct 31 00:07:18.025292 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:18.025272 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5aba07a-x585x"] Oct 31 00:07:18.589060 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:18.589030 7570 generic.go:356] "Generic (PLEG): container finished" podID="60ef3464-bc1e-4efa-b829-7264ef415ab7" containerID="fc525553f7b1be6ea05ee46ca784942ddd063b432968cb540dde06be97080cc1" exitCode=0 Oct 31 00:07:18.589416 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:18.589116 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" event={"ID":"60ef3464-bc1e-4efa-b829-7264ef415ab7","Type":"ContainerDied","Data":"fc525553f7b1be6ea05ee46ca784942ddd063b432968cb540dde06be97080cc1"} Oct 31 00:07:18.589416 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:18.589185 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" event={"ID":"60ef3464-bc1e-4efa-b829-7264ef415ab7","Type":"ContainerStarted","Data":"da546cfeed19b143043b2a5348c996dcafcd9af328fddf7b7101e5d95bc06891"} Oct 31 00:07:18.601429 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:18.601410 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d36cc25-4820-4388-8e5d-a512071934d3" path="/var/lib/kubelet/pods/6d36cc25-4820-4388-8e5d-a512071934d3/volumes" Oct 31 00:07:19.736908 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:19.736885 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:19.810672 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:19.810646 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-xks-ca\") pod \"60ef3464-bc1e-4efa-b829-7264ef415ab7\" (UID: \"60ef3464-bc1e-4efa-b829-7264ef415ab7\") " Oct 31 00:07:19.810831 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:19.810706 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ef3464-bc1e-4efa-b829-7264ef415ab7-kube-api-access-7ckpv\" (UniqueName: \"kubernetes.io/projected/60ef3464-bc1e-4efa-b829-7264ef415ab7-kube-api-access-7ckpv\") pod \"60ef3464-bc1e-4efa-b829-7264ef415ab7\" (UID: \"60ef3464-bc1e-4efa-b829-7264ef415ab7\") " Oct 31 00:07:19.812871 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:19.812848 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ef3464-bc1e-4efa-b829-7264ef415ab7-kube-api-access-7ckpv" pod "60ef3464-bc1e-4efa-b829-7264ef415ab7" (UID: "60ef3464-bc1e-4efa-b829-7264ef415ab7"). InnerVolumeSpecName "kube-api-access-7ckpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:19.812961 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:19.812848 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-xks-ca" pod "60ef3464-bc1e-4efa-b829-7264ef415ab7" (UID: "60ef3464-bc1e-4efa-b829-7264ef415ab7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:19.911294 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:19.911225 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-openshift-ca\") pod \"60ef3464-bc1e-4efa-b829-7264ef415ab7\" (UID: \"60ef3464-bc1e-4efa-b829-7264ef415ab7\") " Oct 31 00:07:19.911608 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:19.911580 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-openshift-ca" pod "60ef3464-bc1e-4efa-b829-7264ef415ab7" (UID: "60ef3464-bc1e-4efa-b829-7264ef415ab7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:19.911721 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:19.911655 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:19.911721 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:19.911680 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ckpv\" (UniqueName: \"kubernetes.io/projected/60ef3464-bc1e-4efa-b829-7264ef415ab7-kube-api-access-7ckpv\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:20.012158 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:20.012131 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ef3464-bc1e-4efa-b829-7264ef415ab7-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:20.601942 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:20.601914 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" Oct 31 00:07:20.602107 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:20.601961 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn" event={"ID":"60ef3464-bc1e-4efa-b829-7264ef415ab7","Type":"ContainerDied","Data":"da546cfeed19b143043b2a5348c996dcafcd9af328fddf7b7101e5d95bc06891"} Oct 31 00:07:20.602107 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:20.602014 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da546cfeed19b143043b2a5348c996dcafcd9af328fddf7b7101e5d95bc06891" Oct 31 00:07:27.860132 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.860100 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t"] Oct 31 00:07:27.860572 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.860559 7570 cpu_manager.go:414] "containerMap: removing container" podUID="60ef3464-bc1e-4efa-b829-7264ef415ab7" containerName="revoke-keys" Oct 31 00:07:27.860617 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.860573 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ef3464-bc1e-4efa-b829-7264ef415ab7" containerName="revoke-keys" Oct 31 00:07:27.860663 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.860654 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ef3464-bc1e-4efa-b829-7264ef415ab7" containerName="revoke-keys" Oct 31 00:07:27.864069 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.864050 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:27.869889 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.869862 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:07:27.870676 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.870653 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:07:27.870845 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.870664 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:07:27.870845 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.870786 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:07:27.874089 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.874070 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t"] Oct 31 00:07:27.980559 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.980527 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rlqk\" (UniqueName: \"kubernetes.io/projected/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-kube-api-access-9rlqk\") pod \"maas-api-revoke-sub-3d4156f1-47l6t\" (UID: \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:27.980559 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.980572 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-47l6t\" (UID: \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:27.980767 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:27.980702 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-47l6t\" (UID: \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:28.081866 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:28.081836 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rlqk\" (UniqueName: \"kubernetes.io/projected/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-kube-api-access-9rlqk\") pod \"maas-api-revoke-sub-3d4156f1-47l6t\" (UID: \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:28.081866 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:28.081867 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-47l6t\" (UID: \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:28.082043 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:28.081910 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-47l6t\" (UID: \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:28.082571 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:28.082542 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-47l6t\" (UID: \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:28.084346 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:28.084326 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-47l6t\" (UID: \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:28.094222 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:28.094202 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rlqk\" (UniqueName: \"kubernetes.io/projected/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-kube-api-access-9rlqk\") pod \"maas-api-revoke-sub-3d4156f1-47l6t\" (UID: \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:28.179529 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:28.179450 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:28.474943 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:07:28.474910 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ff3e36_1820_4b0a_8ae3_bf4f8546e48c.slice/crio-1a61143c43bc8cb66d07eaf72bf2cf10582fd7cef4f03e6a3b84ebdcf4155b5e WatchSource:0}: Error finding container 1a61143c43bc8cb66d07eaf72bf2cf10582fd7cef4f03e6a3b84ebdcf4155b5e: Status 404 returned error can't find the container with id 1a61143c43bc8cb66d07eaf72bf2cf10582fd7cef4f03e6a3b84ebdcf4155b5e Oct 31 00:07:28.475533 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:28.475505 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t"] Oct 31 00:07:28.652277 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:28.652236 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" event={"ID":"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c","Type":"ContainerStarted","Data":"1a61143c43bc8cb66d07eaf72bf2cf10582fd7cef4f03e6a3b84ebdcf4155b5e"} Oct 31 00:07:29.410725 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:07:29.410692 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:07:29.659761 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:29.659730 7570 generic.go:356] "Generic (PLEG): container finished" podID="f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c" containerID="836eab80ee4a2571f478fdf97b85d2a7a2e9a21bd2e4b72d58537c68dbb332dd" exitCode=0 Oct 31 00:07:29.659914 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:29.659817 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" event={"ID":"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c","Type":"ContainerDied","Data":"836eab80ee4a2571f478fdf97b85d2a7a2e9a21bd2e4b72d58537c68dbb332dd"} Oct 31 00:07:30.800900 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:30.800877 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:30.909105 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:30.909076 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-xks-ca\") pod \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\" (UID: \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\") " Oct 31 00:07:30.909261 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:30.909153 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-openshift-ca\") pod \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\" (UID: \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\") " Oct 31 00:07:30.909261 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:30.909199 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-kube-api-access-9rlqk\" (UniqueName: \"kubernetes.io/projected/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-kube-api-access-9rlqk\") pod \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\" (UID: \"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c\") " Oct 31 00:07:30.909531 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:30.909510 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-openshift-ca" pod "f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c" (UID: "f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:30.911314 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:30.911279 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-xks-ca" pod "f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c" (UID: "f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:30.911440 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:30.911420 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-kube-api-access-9rlqk" pod "f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c" (UID: "f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c"). InnerVolumeSpecName "kube-api-access-9rlqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:31.010734 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:31.010711 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:31.010815 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:31.010737 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:31.010815 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:31.010747 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rlqk\" (UniqueName: \"kubernetes.io/projected/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c-kube-api-access-9rlqk\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:31.672529 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:31.672500 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" Oct 31 00:07:31.672529 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:31.672508 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t" event={"ID":"f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c","Type":"ContainerDied","Data":"1a61143c43bc8cb66d07eaf72bf2cf10582fd7cef4f03e6a3b84ebdcf4155b5e"} Oct 31 00:07:31.672771 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:31.672549 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a61143c43bc8cb66d07eaf72bf2cf10582fd7cef4f03e6a3b84ebdcf4155b5e" Oct 31 00:07:43.563413 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:43.563385 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log" Oct 31 00:07:44.030938 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:44.030904 7570 scope.go:122] "RemoveContainer" containerID="f74018888a3f838d1480197109c8cb52c10f5ef0833e4cb163dce688a93659e1" Oct 31 00:07:44.062758 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:44.062736 7570 scope.go:122] "RemoveContainer" containerID="58f6be62a832dd344eb19ea039c4c9fe57e3663e1acd371b9a3bc84465006187" Oct 31 00:07:44.075515 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:44.075497 7570 scope.go:122] "RemoveContainer" containerID="218239c48ae3a7889de183724fc116f117419a6339706ca8313b4216a048c310" Oct 31 00:07:44.089932 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:44.089908 7570 scope.go:122] "RemoveContainer" containerID="66f8a656b3c8f7d5b9ae5d9e1d1fc8b464e1259845fa619d4bb353f63cd80d96" Oct 31 00:07:44.125488 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:44.125374 7570 scope.go:122] "RemoveContainer" containerID="9964ef3208428295b2be8c0118a49764128634711070f478dadbb40176e57fe6" Oct 31 00:07:44.138866 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:44.138779 7570 scope.go:122] "RemoveContainer" containerID="43264f2db4aedb9084176733efe7533377dff1dcb6a22d7d4869e437c4693da3" Oct 31 00:07:44.151616 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:44.151516 7570 scope.go:122] "RemoveContainer" containerID="5876d9f97e9a3a4ba40ef8652bca05ec8e8328294e24a88c46625df1f944cb70" Oct 31 00:07:44.192642 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:44.192607 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p5kqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:07:44.192750 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:44.192676 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p5kqz" podUID="9a41f90f-265b-4a47-a2bc-9239bbd1822b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:07:45.569125 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:45.569019 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log" Oct 31 00:07:49.244243 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.244214 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t"] Oct 31 00:07:49.249034 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.249007 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-47l6t"] Oct 31 00:07:49.269549 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.269523 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch"] Oct 31 00:07:49.269997 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.269983 7570 cpu_manager.go:414] "containerMap: removing container" podUID="f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c" containerName="revoke-keys" Oct 31 00:07:49.270045 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.269997 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c" containerName="revoke-keys" Oct 31 00:07:49.270080 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.270063 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c" containerName="revoke-keys" Oct 31 00:07:49.273290 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.273274 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:49.279620 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.279599 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:07:49.281633 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.281610 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:07:49.281772 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.281652 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:07:49.282336 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.282245 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:07:49.283368 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.283345 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch"] Oct 31 00:07:49.322831 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.322798 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsgjf\" (UniqueName: \"kubernetes.io/projected/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-kube-api-access-zsgjf\") pod \"maas-api-revoke-sub-3d4156f1-8tkch\" (UID: \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:49.322967 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.322861 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8tkch\" (UID: \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:49.323028 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.323003 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8tkch\" (UID: \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:49.423844 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.423812 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsgjf\" (UniqueName: \"kubernetes.io/projected/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-kube-api-access-zsgjf\") pod \"maas-api-revoke-sub-3d4156f1-8tkch\" (UID: \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:49.424018 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.423881 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8tkch\" (UID: \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:49.424018 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.423936 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8tkch\" (UID: \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:49.424537 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.424512 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8tkch\" (UID: \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:49.426270 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.426248 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8tkch\" (UID: \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:49.435875 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.435847 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsgjf\" (UniqueName: \"kubernetes.io/projected/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-kube-api-access-zsgjf\") pod \"maas-api-revoke-sub-3d4156f1-8tkch\" (UID: \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:49.590221 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.590196 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:49.866837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:49.866801 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch"] Oct 31 00:07:49.868717 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:07:49.868686 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8322c5b_f947_48e5_b3f5_2ef597bc5ba1.slice/crio-918c9a5d37fc152989d09f1bf87b952b6c8334deeaf27589728a784442c654be WatchSource:0}: Error finding container 918c9a5d37fc152989d09f1bf87b952b6c8334deeaf27589728a784442c654be: Status 404 returned error can't find the container with id 918c9a5d37fc152989d09f1bf87b952b6c8334deeaf27589728a784442c654be Oct 31 00:07:50.600936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:50.600905 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c" path="/var/lib/kubelet/pods/f1ff3e36-1820-4b0a-8ae3-bf4f8546e48c/volumes" Oct 31 00:07:50.808270 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:50.808243 7570 generic.go:356] "Generic (PLEG): container finished" podID="d8322c5b-f947-48e5-b3f5-2ef597bc5ba1" containerID="fb3a5cf59f673390cb2f08e97080ecdef3237dd3eee4df37a259ae53d1a1e2e2" exitCode=0 Oct 31 00:07:50.808400 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:50.808338 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" event={"ID":"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1","Type":"ContainerDied","Data":"fb3a5cf59f673390cb2f08e97080ecdef3237dd3eee4df37a259ae53d1a1e2e2"} Oct 31 00:07:50.808400 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:50.808388 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" event={"ID":"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1","Type":"ContainerStarted","Data":"918c9a5d37fc152989d09f1bf87b952b6c8334deeaf27589728a784442c654be"} Oct 31 00:07:51.946400 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:51.946377 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:52.049134 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:52.049108 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-xks-ca\") pod \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\" (UID: \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\") " Oct 31 00:07:52.049276 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:52.049192 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-openshift-ca\") pod \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\" (UID: \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\") " Oct 31 00:07:52.049350 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:52.049327 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-kube-api-access-zsgjf\" (UniqueName: \"kubernetes.io/projected/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-kube-api-access-zsgjf\") pod \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\" (UID: \"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1\") " Oct 31 00:07:52.049595 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:52.049562 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-openshift-ca" pod "d8322c5b-f947-48e5-b3f5-2ef597bc5ba1" (UID: "d8322c5b-f947-48e5-b3f5-2ef597bc5ba1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:52.049710 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:52.049674 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:52.051447 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:52.051416 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-xks-ca" pod "d8322c5b-f947-48e5-b3f5-2ef597bc5ba1" (UID: "d8322c5b-f947-48e5-b3f5-2ef597bc5ba1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:52.051575 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:52.051490 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-kube-api-access-zsgjf" pod "d8322c5b-f947-48e5-b3f5-2ef597bc5ba1" (UID: "d8322c5b-f947-48e5-b3f5-2ef597bc5ba1"). InnerVolumeSpecName "kube-api-access-zsgjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:52.150625 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:52.150599 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsgjf\" (UniqueName: \"kubernetes.io/projected/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-kube-api-access-zsgjf\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:52.150625 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:52.150625 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:52.820892 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:52.820863 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" Oct 31 00:07:52.821050 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:52.820906 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch" event={"ID":"d8322c5b-f947-48e5-b3f5-2ef597bc5ba1","Type":"ContainerDied","Data":"918c9a5d37fc152989d09f1bf87b952b6c8334deeaf27589728a784442c654be"} Oct 31 00:07:52.821050 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:07:52.820955 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="918c9a5d37fc152989d09f1bf87b952b6c8334deeaf27589728a784442c654be" Oct 31 00:07:59.411825 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:07:59.411786 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:08:14.922307 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:14.922227 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk"] Oct 31 00:08:14.922721 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:14.922684 7570 cpu_manager.go:414] "containerMap: removing container" podUID="d8322c5b-f947-48e5-b3f5-2ef597bc5ba1" containerName="revoke-keys" Oct 31 00:08:14.922721 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:14.922699 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8322c5b-f947-48e5-b3f5-2ef597bc5ba1" containerName="revoke-keys" Oct 31 00:08:14.922827 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:14.922814 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8322c5b-f947-48e5-b3f5-2ef597bc5ba1" containerName="revoke-keys" Oct 31 00:08:14.926604 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:14.926579 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:14.932772 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:14.932748 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:08:14.932907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:14.932798 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:08:14.932907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:14.932795 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:08:14.932907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:14.932798 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:08:14.939273 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:14.939245 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk"] Oct 31 00:08:15.088304 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:15.088271 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-mrsbk\" (UID: \"c4cd85c7-3856-436e-a326-d000dcc14e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:15.088484 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:15.088399 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-mrsbk\" (UID: \"c4cd85c7-3856-436e-a326-d000dcc14e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:15.088484 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:15.088426 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6p6n\" (UniqueName: \"kubernetes.io/projected/c4cd85c7-3856-436e-a326-d000dcc14e5a-kube-api-access-n6p6n\") pod \"maas-api-revoke-sub-e2fc9c5b-mrsbk\" (UID: \"c4cd85c7-3856-436e-a326-d000dcc14e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:15.189110 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:15.189031 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-mrsbk\" (UID: \"c4cd85c7-3856-436e-a326-d000dcc14e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:15.189110 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:15.189099 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-mrsbk\" (UID: \"c4cd85c7-3856-436e-a326-d000dcc14e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:15.189317 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:15.189117 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6p6n\" (UniqueName: \"kubernetes.io/projected/c4cd85c7-3856-436e-a326-d000dcc14e5a-kube-api-access-n6p6n\") pod \"maas-api-revoke-sub-e2fc9c5b-mrsbk\" (UID: \"c4cd85c7-3856-436e-a326-d000dcc14e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:15.189753 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:15.189729 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-mrsbk\" (UID: \"c4cd85c7-3856-436e-a326-d000dcc14e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:15.191630 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:15.191612 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-mrsbk\" (UID: \"c4cd85c7-3856-436e-a326-d000dcc14e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:15.201811 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:15.201792 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6p6n\" (UniqueName: \"kubernetes.io/projected/c4cd85c7-3856-436e-a326-d000dcc14e5a-kube-api-access-n6p6n\") pod \"maas-api-revoke-sub-e2fc9c5b-mrsbk\" (UID: \"c4cd85c7-3856-436e-a326-d000dcc14e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:15.242368 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:15.242343 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:15.521025 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:15.521002 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk"] Oct 31 00:08:15.522814 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:08:15.522787 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4cd85c7_3856_436e_a326_d000dcc14e5a.slice/crio-43f1e25b58a862bd2de786c7d404bf1a7b92c5169d0a112dc5e10d65587e6dc9 WatchSource:0}: Error finding container 43f1e25b58a862bd2de786c7d404bf1a7b92c5169d0a112dc5e10d65587e6dc9: Status 404 returned error can't find the container with id 43f1e25b58a862bd2de786c7d404bf1a7b92c5169d0a112dc5e10d65587e6dc9 Oct 31 00:08:16.008523 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:16.008454 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" event={"ID":"c4cd85c7-3856-436e-a326-d000dcc14e5a","Type":"ContainerStarted","Data":"181c9a34794ea7d9bb1c80eaef83da93dbb26b6a8f0205dc658456cc6491d888"} Oct 31 00:08:16.008961 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:16.008538 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" event={"ID":"c4cd85c7-3856-436e-a326-d000dcc14e5a","Type":"ContainerStarted","Data":"43f1e25b58a862bd2de786c7d404bf1a7b92c5169d0a112dc5e10d65587e6dc9"} Oct 31 00:08:16.029131 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:16.029082 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" podStartSLOduration=2.029070511 podStartE2EDuration="2.029070511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:16.027025192 +0000 UTC m=+2907.546026282" watchObservedRunningTime="2026-10-31 00:08:16.029070511 +0000 UTC m=+2907.548071602" Oct 31 00:08:17.015805 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:17.015772 7570 generic.go:356] "Generic (PLEG): container finished" podID="c4cd85c7-3856-436e-a326-d000dcc14e5a" containerID="181c9a34794ea7d9bb1c80eaef83da93dbb26b6a8f0205dc658456cc6491d888" exitCode=0 Oct 31 00:08:17.016329 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:17.015850 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" event={"ID":"c4cd85c7-3856-436e-a326-d000dcc14e5a","Type":"ContainerDied","Data":"181c9a34794ea7d9bb1c80eaef83da93dbb26b6a8f0205dc658456cc6491d888"} Oct 31 00:08:18.163337 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:18.163308 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:18.319716 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:18.319648 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-openshift-ca\") pod \"c4cd85c7-3856-436e-a326-d000dcc14e5a\" (UID: \"c4cd85c7-3856-436e-a326-d000dcc14e5a\") " Oct 31 00:08:18.319716 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:18.319717 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4cd85c7-3856-436e-a326-d000dcc14e5a-kube-api-access-n6p6n\" (UniqueName: \"kubernetes.io/projected/c4cd85c7-3856-436e-a326-d000dcc14e5a-kube-api-access-n6p6n\") pod \"c4cd85c7-3856-436e-a326-d000dcc14e5a\" (UID: \"c4cd85c7-3856-436e-a326-d000dcc14e5a\") " Oct 31 00:08:18.319883 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:18.319780 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-xks-ca\") pod \"c4cd85c7-3856-436e-a326-d000dcc14e5a\" (UID: \"c4cd85c7-3856-436e-a326-d000dcc14e5a\") " Oct 31 00:08:18.320077 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:18.320051 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-openshift-ca" pod "c4cd85c7-3856-436e-a326-d000dcc14e5a" (UID: "c4cd85c7-3856-436e-a326-d000dcc14e5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:18.322019 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:18.321993 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-xks-ca" pod "c4cd85c7-3856-436e-a326-d000dcc14e5a" (UID: "c4cd85c7-3856-436e-a326-d000dcc14e5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:18.322076 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:18.322022 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4cd85c7-3856-436e-a326-d000dcc14e5a-kube-api-access-n6p6n" pod "c4cd85c7-3856-436e-a326-d000dcc14e5a" (UID: "c4cd85c7-3856-436e-a326-d000dcc14e5a"). InnerVolumeSpecName "kube-api-access-n6p6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:18.421181 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:18.421157 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:18.421181 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:18.421178 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6p6n\" (UniqueName: \"kubernetes.io/projected/c4cd85c7-3856-436e-a326-d000dcc14e5a-kube-api-access-n6p6n\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:18.421326 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:18.421190 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cd85c7-3856-436e-a326-d000dcc14e5a-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:19.028158 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:19.028079 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" Oct 31 00:08:19.028158 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:19.028114 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk" event={"ID":"c4cd85c7-3856-436e-a326-d000dcc14e5a","Type":"ContainerDied","Data":"43f1e25b58a862bd2de786c7d404bf1a7b92c5169d0a112dc5e10d65587e6dc9"} Oct 31 00:08:19.028158 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:19.028159 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43f1e25b58a862bd2de786c7d404bf1a7b92c5169d0a112dc5e10d65587e6dc9" Oct 31 00:08:20.042308 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:20.042275 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv"] Oct 31 00:08:20.052359 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:20.052327 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5510eb64-dmzxv"] Oct 31 00:08:20.601207 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:20.601173 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53048d8-1746-42ae-91a4-f160a2948c62" path="/var/lib/kubelet/pods/c53048d8-1746-42ae-91a4-f160a2948c62/volumes" Oct 31 00:08:25.145559 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.145528 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt"] Oct 31 00:08:25.145984 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.145970 7570 cpu_manager.go:414] "containerMap: removing container" podUID="c4cd85c7-3856-436e-a326-d000dcc14e5a" containerName="revoke-keys" Oct 31 00:08:25.146027 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.145985 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4cd85c7-3856-436e-a326-d000dcc14e5a" containerName="revoke-keys" Oct 31 00:08:25.146065 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.146057 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4cd85c7-3856-436e-a326-d000dcc14e5a" containerName="revoke-keys" Oct 31 00:08:25.150552 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.150531 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:25.154981 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.154622 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:08:25.155711 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.155688 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:08:25.155827 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.155738 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:08:25.156292 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.156273 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:08:25.159018 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.158768 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt"] Oct 31 00:08:25.284139 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.284112 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-rxgdt\" (UID: \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:25.284264 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.284164 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-rxgdt\" (UID: \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:25.284308 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.284281 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2d65\" (UniqueName: \"kubernetes.io/projected/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-kube-api-access-s2d65\") pod \"maas-api-revoke-sub-5a29efb3-rxgdt\" (UID: \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:25.385495 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.385446 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-rxgdt\" (UID: \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:25.385606 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.385537 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2d65\" (UniqueName: \"kubernetes.io/projected/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-kube-api-access-s2d65\") pod \"maas-api-revoke-sub-5a29efb3-rxgdt\" (UID: \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:25.385647 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.385606 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-rxgdt\" (UID: \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:25.386285 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.386203 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-rxgdt\" (UID: \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:25.388005 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.387979 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-rxgdt\" (UID: \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:25.398646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.398595 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2d65\" (UniqueName: \"kubernetes.io/projected/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-kube-api-access-s2d65\") pod \"maas-api-revoke-sub-5a29efb3-rxgdt\" (UID: \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:25.466719 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.466696 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:25.747705 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:25.747671 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt"] Oct 31 00:08:25.748504 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:08:25.748473 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e70e07_c686_4ca0_94e0_b8b37a3d2058.slice/crio-a04947b3a88a2a1e2d680560a0fde60d86833de778bd145b72ab4ef9b8ad1f8c WatchSource:0}: Error finding container a04947b3a88a2a1e2d680560a0fde60d86833de778bd145b72ab4ef9b8ad1f8c: Status 404 returned error can't find the container with id a04947b3a88a2a1e2d680560a0fde60d86833de778bd145b72ab4ef9b8ad1f8c Oct 31 00:08:26.069049 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:26.069007 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" event={"ID":"c4e70e07-c686-4ca0-94e0-b8b37a3d2058","Type":"ContainerStarted","Data":"1ca607f5cdcf10104bb0d64c3af48e2992ab26c9f64dd614e2774abfd6de041f"} Oct 31 00:08:26.069212 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:26.069066 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" event={"ID":"c4e70e07-c686-4ca0-94e0-b8b37a3d2058","Type":"ContainerStarted","Data":"a04947b3a88a2a1e2d680560a0fde60d86833de778bd145b72ab4ef9b8ad1f8c"} Oct 31 00:08:26.091067 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:26.091022 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" podStartSLOduration=1.091008754 podStartE2EDuration="1.091008754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:26.087495977 +0000 UTC m=+2917.606497071" watchObservedRunningTime="2026-10-31 00:08:26.091008754 +0000 UTC m=+2917.610009845" Oct 31 00:08:27.079416 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:27.079371 7570 generic.go:356] "Generic (PLEG): container finished" podID="c4e70e07-c686-4ca0-94e0-b8b37a3d2058" containerID="1ca607f5cdcf10104bb0d64c3af48e2992ab26c9f64dd614e2774abfd6de041f" exitCode=0 Oct 31 00:08:27.079869 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:27.079450 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" event={"ID":"c4e70e07-c686-4ca0-94e0-b8b37a3d2058","Type":"ContainerDied","Data":"1ca607f5cdcf10104bb0d64c3af48e2992ab26c9f64dd614e2774abfd6de041f"} Oct 31 00:08:28.490877 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:28.490849 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:28.670512 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:28.670441 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-kube-api-access-s2d65\" (UniqueName: \"kubernetes.io/projected/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-kube-api-access-s2d65\") pod \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\" (UID: \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\") " Oct 31 00:08:28.670823 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:28.670791 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-openshift-ca\") pod \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\" (UID: \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\") " Oct 31 00:08:28.670954 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:28.670863 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-xks-ca\") pod \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\" (UID: \"c4e70e07-c686-4ca0-94e0-b8b37a3d2058\") " Oct 31 00:08:28.671727 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:28.671173 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-openshift-ca" pod "c4e70e07-c686-4ca0-94e0-b8b37a3d2058" (UID: "c4e70e07-c686-4ca0-94e0-b8b37a3d2058"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:28.671727 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:28.671355 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:28.673729 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:28.673704 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-xks-ca" pod "c4e70e07-c686-4ca0-94e0-b8b37a3d2058" (UID: "c4e70e07-c686-4ca0-94e0-b8b37a3d2058"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:28.673837 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:28.673800 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-kube-api-access-s2d65" pod "c4e70e07-c686-4ca0-94e0-b8b37a3d2058" (UID: "c4e70e07-c686-4ca0-94e0-b8b37a3d2058"). InnerVolumeSpecName "kube-api-access-s2d65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:28.772632 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:28.772549 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:28.772632 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:28.772577 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2d65\" (UniqueName: \"kubernetes.io/projected/c4e70e07-c686-4ca0-94e0-b8b37a3d2058-kube-api-access-s2d65\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:29.092394 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:29.092373 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" Oct 31 00:08:29.092549 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:29.092388 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt" event={"ID":"c4e70e07-c686-4ca0-94e0-b8b37a3d2058","Type":"ContainerDied","Data":"a04947b3a88a2a1e2d680560a0fde60d86833de778bd145b72ab4ef9b8ad1f8c"} Oct 31 00:08:29.092549 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:29.092431 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a04947b3a88a2a1e2d680560a0fde60d86833de778bd145b72ab4ef9b8ad1f8c" Oct 31 00:08:29.411031 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:08:29.410944 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:08:41.108052 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.108017 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch"] Oct 31 00:08:41.111420 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.111397 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8tkch"] Oct 31 00:08:41.134232 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.134207 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5"] Oct 31 00:08:41.134653 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.134641 7570 cpu_manager.go:414] "containerMap: removing container" podUID="c4e70e07-c686-4ca0-94e0-b8b37a3d2058" containerName="revoke-keys" Oct 31 00:08:41.134709 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.134656 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e70e07-c686-4ca0-94e0-b8b37a3d2058" containerName="revoke-keys" Oct 31 00:08:41.134767 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.134758 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e70e07-c686-4ca0-94e0-b8b37a3d2058" containerName="revoke-keys" Oct 31 00:08:41.138438 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.138424 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:41.145791 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.145539 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:08:41.147237 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.147216 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:08:41.147666 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.147639 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:08:41.150493 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.150444 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5"] Oct 31 00:08:41.181444 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.181406 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpfmk\" (UniqueName: \"kubernetes.io/projected/809bb7bf-e365-4014-a8dc-d05dd97fc155-kube-api-access-dpfmk\") pod \"maas-api-revoke-sub-3d4156f1-swtz5\" (UID: \"809bb7bf-e365-4014-a8dc-d05dd97fc155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:41.181627 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.181488 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-swtz5\" (UID: \"809bb7bf-e365-4014-a8dc-d05dd97fc155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:41.181691 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.181665 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-swtz5\" (UID: \"809bb7bf-e365-4014-a8dc-d05dd97fc155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:41.246181 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.246151 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:08:41.282664 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.282640 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-swtz5\" (UID: \"809bb7bf-e365-4014-a8dc-d05dd97fc155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:41.282783 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.282700 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpfmk\" (UniqueName: \"kubernetes.io/projected/809bb7bf-e365-4014-a8dc-d05dd97fc155-kube-api-access-dpfmk\") pod \"maas-api-revoke-sub-3d4156f1-swtz5\" (UID: \"809bb7bf-e365-4014-a8dc-d05dd97fc155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:41.282783 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.282737 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-swtz5\" (UID: \"809bb7bf-e365-4014-a8dc-d05dd97fc155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:41.283368 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.283349 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-swtz5\" (UID: \"809bb7bf-e365-4014-a8dc-d05dd97fc155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:41.285170 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.285149 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-swtz5\" (UID: \"809bb7bf-e365-4014-a8dc-d05dd97fc155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:41.303797 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.303769 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpfmk\" (UniqueName: \"kubernetes.io/projected/809bb7bf-e365-4014-a8dc-d05dd97fc155-kube-api-access-dpfmk\") pod \"maas-api-revoke-sub-3d4156f1-swtz5\" (UID: \"809bb7bf-e365-4014-a8dc-d05dd97fc155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:41.453984 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.453920 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:41.731117 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:41.731092 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5"] Oct 31 00:08:41.732603 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:08:41.732579 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809bb7bf_e365_4014_a8dc_d05dd97fc155.slice/crio-96023737cd8a92a781c600aa82c872cdd463733799c0b6efb633b95f80c7c0d9 WatchSource:0}: Error finding container 96023737cd8a92a781c600aa82c872cdd463733799c0b6efb633b95f80c7c0d9: Status 404 returned error can't find the container with id 96023737cd8a92a781c600aa82c872cdd463733799c0b6efb633b95f80c7c0d9 Oct 31 00:08:42.170705 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:42.170668 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" event={"ID":"809bb7bf-e365-4014-a8dc-d05dd97fc155","Type":"ContainerStarted","Data":"20d7b1fdd89a654b2a4dc5a8e7b33ccc32dae2a2584403afb5d83020f23ea0cf"} Oct 31 00:08:42.171098 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:42.170721 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" event={"ID":"809bb7bf-e365-4014-a8dc-d05dd97fc155","Type":"ContainerStarted","Data":"96023737cd8a92a781c600aa82c872cdd463733799c0b6efb633b95f80c7c0d9"} Oct 31 00:08:42.197870 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:42.197820 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" podStartSLOduration=1.197807332 podStartE2EDuration="1.197807332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:42.194803597 +0000 UTC m=+2933.713804686" watchObservedRunningTime="2026-10-31 00:08:42.197807332 +0000 UTC m=+2933.716808466" Oct 31 00:08:42.602016 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:42.601990 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8322c5b-f947-48e5-b3f5-2ef597bc5ba1" path="/var/lib/kubelet/pods/d8322c5b-f947-48e5-b3f5-2ef597bc5ba1/volumes" Oct 31 00:08:43.179355 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:43.179313 7570 generic.go:356] "Generic (PLEG): container finished" podID="809bb7bf-e365-4014-a8dc-d05dd97fc155" containerID="20d7b1fdd89a654b2a4dc5a8e7b33ccc32dae2a2584403afb5d83020f23ea0cf" exitCode=0 Oct 31 00:08:43.179753 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:43.179392 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" event={"ID":"809bb7bf-e365-4014-a8dc-d05dd97fc155","Type":"ContainerDied","Data":"20d7b1fdd89a654b2a4dc5a8e7b33ccc32dae2a2584403afb5d83020f23ea0cf"} Oct 31 00:08:44.214386 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.214351 7570 scope.go:122] "RemoveContainer" containerID="d5a827ee07aaeaae63dcc7224ff9061110146fb94db7c3ae5515a95afb9c488d" Oct 31 00:08:44.225878 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.225855 7570 scope.go:122] "RemoveContainer" containerID="e0a2a5647b3290be3272d54f372d86b760ab4b85af94f2e3f1673f340b46f4c4" Oct 31 00:08:44.236070 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.236046 7570 scope.go:122] "RemoveContainer" containerID="f8378e18462d840aa7fab94c24fbfad1907786111a3722693cf6639b044f0618" Oct 31 00:08:44.333133 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.333111 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:44.414359 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.414325 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809bb7bf-e365-4014-a8dc-d05dd97fc155-kube-api-access-dpfmk\" (UniqueName: \"kubernetes.io/projected/809bb7bf-e365-4014-a8dc-d05dd97fc155-kube-api-access-dpfmk\") pod \"809bb7bf-e365-4014-a8dc-d05dd97fc155\" (UID: \"809bb7bf-e365-4014-a8dc-d05dd97fc155\") " Oct 31 00:08:44.414580 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.414514 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-xks-ca\") pod \"809bb7bf-e365-4014-a8dc-d05dd97fc155\" (UID: \"809bb7bf-e365-4014-a8dc-d05dd97fc155\") " Oct 31 00:08:44.414652 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.414620 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-openshift-ca\") pod \"809bb7bf-e365-4014-a8dc-d05dd97fc155\" (UID: \"809bb7bf-e365-4014-a8dc-d05dd97fc155\") " Oct 31 00:08:44.415042 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.415002 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-openshift-ca" pod "809bb7bf-e365-4014-a8dc-d05dd97fc155" (UID: "809bb7bf-e365-4014-a8dc-d05dd97fc155"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:44.416770 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.416743 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809bb7bf-e365-4014-a8dc-d05dd97fc155-kube-api-access-dpfmk" pod "809bb7bf-e365-4014-a8dc-d05dd97fc155" (UID: "809bb7bf-e365-4014-a8dc-d05dd97fc155"). InnerVolumeSpecName "kube-api-access-dpfmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:44.417205 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.417177 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-xks-ca" pod "809bb7bf-e365-4014-a8dc-d05dd97fc155" (UID: "809bb7bf-e365-4014-a8dc-d05dd97fc155"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:44.515571 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.515508 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:44.515571 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.515535 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809bb7bf-e365-4014-a8dc-d05dd97fc155-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:44.515571 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:44.515547 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpfmk\" (UniqueName: \"kubernetes.io/projected/809bb7bf-e365-4014-a8dc-d05dd97fc155-kube-api-access-dpfmk\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:45.193239 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:45.193202 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" event={"ID":"809bb7bf-e365-4014-a8dc-d05dd97fc155","Type":"ContainerDied","Data":"96023737cd8a92a781c600aa82c872cdd463733799c0b6efb633b95f80c7c0d9"} Oct 31 00:08:45.193239 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:45.193224 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5" Oct 31 00:08:45.193447 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:08:45.193258 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96023737cd8a92a781c600aa82c872cdd463733799c0b6efb633b95f80c7c0d9" Oct 31 00:08:59.411961 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:08:59.411924 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:09:03.048344 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.048312 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9"] Oct 31 00:09:03.048860 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.048843 7570 cpu_manager.go:414] "containerMap: removing container" podUID="809bb7bf-e365-4014-a8dc-d05dd97fc155" containerName="revoke-keys" Oct 31 00:09:03.048969 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.048863 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809bb7bf-e365-4014-a8dc-d05dd97fc155" containerName="revoke-keys" Oct 31 00:09:03.048969 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.048962 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="809bb7bf-e365-4014-a8dc-d05dd97fc155" containerName="revoke-keys" Oct 31 00:09:03.052527 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.052511 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.061744 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.061721 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-93f1ad-openshift-default-dockercfg-wmj4c\"" Oct 31 00:09:03.071769 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.071748 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9"] Oct 31 00:09:03.142752 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.142720 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-credential-socket\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.142907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.142771 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-socket\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.142907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.142813 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-envoy\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.142990 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.142923 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istiod-ca-cert\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.142990 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.142963 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-podinfo\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.143064 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.143018 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgvqs\" (UniqueName: \"kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-kube-api-access-cgvqs\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.143136 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.143119 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-certs\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.143170 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.143149 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-token\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.143203 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.143170 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-data\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.244495 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.244439 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-credential-socket\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.244685 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.244534 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-socket\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.244685 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.244564 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-envoy\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.244685 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.244611 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istiod-ca-cert\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.244685 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.244657 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-podinfo\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.244904 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.244727 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgvqs\" (UniqueName: \"kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-kube-api-access-cgvqs\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.244904 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.244767 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-certs\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.244904 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.244804 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-token\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.244904 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.244838 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-data\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.245322 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.245296 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-data\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.245641 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.245616 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-credential-socket\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.245869 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.245850 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-socket\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.246415 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.246384 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-certs\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.247657 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.247637 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istiod-ca-cert\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.250986 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.250962 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-podinfo\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.259490 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.259431 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-token\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.259931 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.259908 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgvqs\" (UniqueName: \"kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-kube-api-access-cgvqs\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.260665 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.260646 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-envoy\") pod \"e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.366194 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.366165 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:03.651014 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.650986 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9"] Oct 31 00:09:03.652401 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:09:03.652368 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ae1ff0_9b19_456c_b564_8e21c1b7baf5.slice/crio-24a0a24d3e6769dd70d8682a8ddd4c3d52af55433e06470713c8b89df626248e WatchSource:0}: Error finding container 24a0a24d3e6769dd70d8682a8ddd4c3d52af55433e06470713c8b89df626248e: Status 404 returned error can't find the container with id 24a0a24d3e6769dd70d8682a8ddd4c3d52af55433e06470713c8b89df626248e Oct 31 00:09:03.655037 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.654997 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:09:03.655111 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.655092 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:09:03.655157 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:03.655141 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:09:04.320324 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:04.320289 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" event={"ID":"57ae1ff0-9b19-456c-b564-8e21c1b7baf5","Type":"ContainerStarted","Data":"be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe"} Oct 31 00:09:04.320730 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:04.320352 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" event={"ID":"57ae1ff0-9b19-456c-b564-8e21c1b7baf5","Type":"ContainerStarted","Data":"24a0a24d3e6769dd70d8682a8ddd4c3d52af55433e06470713c8b89df626248e"} Oct 31 00:09:04.347034 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:04.346949 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" podStartSLOduration=1.346936396 podStartE2EDuration="1.346936396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:04.345251148 +0000 UTC m=+2955.864252237" watchObservedRunningTime="2026-10-31 00:09:04.346936396 +0000 UTC m=+2955.865937488" Oct 31 00:09:04.366807 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:04.366777 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:04.371943 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:04.371920 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:05.326839 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:05.326811 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:05.327785 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:05.327767 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:07.960184 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:07.960148 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb"] Oct 31 00:09:07.964285 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:07.964262 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" Oct 31 00:09:07.969826 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:07.969803 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk"] Oct 31 00:09:07.973761 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:07.973743 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" Oct 31 00:09:07.979488 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:07.979450 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb"] Oct 31 00:09:07.987072 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:07.987049 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-93f1ad\"" Oct 31 00:09:07.993683 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:07.993660 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk"] Oct 31 00:09:08.043805 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.043778 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl"] Oct 31 00:09:08.052488 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.052451 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k4qsl"] Oct 31 00:09:08.093991 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.093953 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a782d992-67cc-4c0a-ad76-5652e94178b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk\" (UID: \"a782d992-67cc-4c0a-ad76-5652e94178b1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" Oct 31 00:09:08.094125 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.094036 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79k9\" (UniqueName: \"kubernetes.io/projected/a782d992-67cc-4c0a-ad76-5652e94178b1-kube-api-access-z79k9\") pod \"payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk\" (UID: \"a782d992-67cc-4c0a-ad76-5652e94178b1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" Oct 31 00:09:08.094125 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.094074 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9zxt\" (UniqueName: \"kubernetes.io/projected/c0eb9a33-1845-4dee-aab3-0bc0ed460447-kube-api-access-m9zxt\") pod \"payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb\" (UID: \"c0eb9a33-1845-4dee-aab3-0bc0ed460447\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" Oct 31 00:09:08.094125 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.094095 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0eb9a33-1845-4dee-aab3-0bc0ed460447-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb\" (UID: \"c0eb9a33-1845-4dee-aab3-0bc0ed460447\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" Oct 31 00:09:08.195442 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.195399 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a782d992-67cc-4c0a-ad76-5652e94178b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk\" (UID: \"a782d992-67cc-4c0a-ad76-5652e94178b1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" Oct 31 00:09:08.195645 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.195501 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79k9\" (UniqueName: \"kubernetes.io/projected/a782d992-67cc-4c0a-ad76-5652e94178b1-kube-api-access-z79k9\") pod \"payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk\" (UID: \"a782d992-67cc-4c0a-ad76-5652e94178b1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" Oct 31 00:09:08.195645 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.195539 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9zxt\" (UniqueName: \"kubernetes.io/projected/c0eb9a33-1845-4dee-aab3-0bc0ed460447-kube-api-access-m9zxt\") pod \"payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb\" (UID: \"c0eb9a33-1845-4dee-aab3-0bc0ed460447\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" Oct 31 00:09:08.195645 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.195573 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0eb9a33-1845-4dee-aab3-0bc0ed460447-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb\" (UID: \"c0eb9a33-1845-4dee-aab3-0bc0ed460447\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" Oct 31 00:09:08.196231 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.196205 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a782d992-67cc-4c0a-ad76-5652e94178b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk\" (UID: \"a782d992-67cc-4c0a-ad76-5652e94178b1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" Oct 31 00:09:08.196325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.196243 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0eb9a33-1845-4dee-aab3-0bc0ed460447-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb\" (UID: \"c0eb9a33-1845-4dee-aab3-0bc0ed460447\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" Oct 31 00:09:08.214031 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.213973 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79k9\" (UniqueName: \"kubernetes.io/projected/a782d992-67cc-4c0a-ad76-5652e94178b1-kube-api-access-z79k9\") pod \"payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk\" (UID: \"a782d992-67cc-4c0a-ad76-5652e94178b1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" Oct 31 00:09:08.214270 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.214253 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9zxt\" (UniqueName: \"kubernetes.io/projected/c0eb9a33-1845-4dee-aab3-0bc0ed460447-kube-api-access-m9zxt\") pod \"payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb\" (UID: \"c0eb9a33-1845-4dee-aab3-0bc0ed460447\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" Oct 31 00:09:08.279415 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.279390 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" Oct 31 00:09:08.289679 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.289661 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" Oct 31 00:09:08.591449 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.591425 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb"] Oct 31 00:09:08.593367 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:09:08.593341 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0eb9a33_1845_4dee_aab3_0bc0ed460447.slice/crio-6d1bd8d85f64b80963810ae2b5515f0c0664254fe14267db5dd9677ddc5b3220 WatchSource:0}: Error finding container 6d1bd8d85f64b80963810ae2b5515f0c0664254fe14267db5dd9677ddc5b3220: Status 404 returned error can't find the container with id 6d1bd8d85f64b80963810ae2b5515f0c0664254fe14267db5dd9677ddc5b3220 Oct 31 00:09:08.600276 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:09:08.600244 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda782d992_67cc_4c0a_ad76_5652e94178b1.slice/crio-e861ffea0b209d244f1630dbb9f0d54cffc5beddb7ee4b8872419040c080f6f9 WatchSource:0}: Error finding container e861ffea0b209d244f1630dbb9f0d54cffc5beddb7ee4b8872419040c080f6f9: Status 404 returned error can't find the container with id e861ffea0b209d244f1630dbb9f0d54cffc5beddb7ee4b8872419040c080f6f9 Oct 31 00:09:08.603371 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.603341 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ac295b-1237-47ff-891f-f7974c867c2e" path="/var/lib/kubelet/pods/97ac295b-1237-47ff-891f-f7974c867c2e/volumes" Oct 31 00:09:08.603761 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:08.603744 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk"] Oct 31 00:09:09.356373 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:09.356342 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" event={"ID":"c0eb9a33-1845-4dee-aab3-0bc0ed460447","Type":"ContainerStarted","Data":"0a8cead280b2f7d0e6f907767e70037c3b6889a07544f6cd8ee3f95ee1adc394"} Oct 31 00:09:09.356905 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:09.356392 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" event={"ID":"c0eb9a33-1845-4dee-aab3-0bc0ed460447","Type":"ContainerStarted","Data":"6d1bd8d85f64b80963810ae2b5515f0c0664254fe14267db5dd9677ddc5b3220"} Oct 31 00:09:09.356905 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:09.356456 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" Oct 31 00:09:09.358430 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:09.358407 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" event={"ID":"a782d992-67cc-4c0a-ad76-5652e94178b1","Type":"ContainerStarted","Data":"ea8d5ac2543b19c9c5b5b46982cd94b2ead83dff61b52791f2b3c6c69eaee592"} Oct 31 00:09:09.358569 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:09.358448 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" event={"ID":"a782d992-67cc-4c0a-ad76-5652e94178b1","Type":"ContainerStarted","Data":"e861ffea0b209d244f1630dbb9f0d54cffc5beddb7ee4b8872419040c080f6f9"} Oct 31 00:09:09.358569 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:09.358530 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" Oct 31 00:09:09.379649 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:09.379603 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" podStartSLOduration=2.379587655 podStartE2EDuration="2.379587655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:09.376093282 +0000 UTC m=+2960.895094377" watchObservedRunningTime="2026-10-31 00:09:09.379587655 +0000 UTC m=+2960.898588813" Oct 31 00:09:09.398703 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:09.398651 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" podStartSLOduration=2.398639007 podStartE2EDuration="2.398639007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:09.396707919 +0000 UTC m=+2960.915709012" watchObservedRunningTime="2026-10-31 00:09:09.398639007 +0000 UTC m=+2960.917640096" Oct 31 00:09:20.368723 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:20.368690 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" Oct 31 00:09:20.369537 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:20.369512 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" Oct 31 00:09:20.687042 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:20.687010 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": read tcp 10.134.0.2:36250->10.134.0.223:15021: read: connection reset by peer" start-of-body= Oct 31 00:09:20.687257 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:20.687092 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": read tcp 10.134.0.2:36250->10.134.0.223:15021: read: connection reset by peer" Oct 31 00:09:21.462132 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:21.462092 7570 generic.go:356] "Generic (PLEG): container finished" podID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerID="be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe" exitCode=0 Oct 31 00:09:21.462764 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:21.462149 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" event={"ID":"57ae1ff0-9b19-456c-b564-8e21c1b7baf5","Type":"ContainerDied","Data":"be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe"} Oct 31 00:09:21.462764 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:21.462622 7570 scope.go:122] "RemoveContainer" containerID="be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe" Oct 31 00:09:21.464594 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:21.464560 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:09:21.464715 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:21.464650 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:09:21.464715 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:21.464698 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:09:22.469779 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:22.469747 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" event={"ID":"57ae1ff0-9b19-456c-b564-8e21c1b7baf5","Type":"ContainerStarted","Data":"baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9"} Oct 31 00:09:23.366605 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:23.366571 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:23.366822 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:23.366800 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 31 00:09:23.366872 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:23.366851 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 31 00:09:24.366905 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:24.366873 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 31 00:09:24.367320 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:24.366932 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 31 00:09:25.367211 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:25.367174 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 31 00:09:25.367749 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:25.367236 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 31 00:09:26.367195 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:26.367164 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 31 00:09:26.367384 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:26.367214 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 31 00:09:27.367188 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:27.367153 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 31 00:09:27.367372 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:27.367205 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 31 00:09:28.370683 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:28.370652 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:28.371157 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:28.371115 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:28.371801 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:28.371778 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:09:29.411859 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:09:29.411809 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:09:44.275807 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:44.275735 7570 scope.go:122] "RemoveContainer" containerID="e6d491fa38ada313319950abb7dee727eb256cbbb499627f05a6b20d5ad59f01" Oct 31 00:09:44.285051 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:44.285032 7570 scope.go:122] "RemoveContainer" containerID="116039e34d9c351dc9ec13055520dda8c46a27a7624044cc42d6338aaee55978" Oct 31 00:09:44.293849 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:44.293829 7570 scope.go:122] "RemoveContainer" containerID="3997980767aebc5329d79db450bf6df6bfb0528eec37ab42b3f3163ffa96a879" Oct 31 00:09:48.531591 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.531559 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j"] Oct 31 00:09:48.535337 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.535313 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:48.541784 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.541751 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:09:48.542188 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.542162 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:09:48.542305 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.542236 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:09:48.542370 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.542314 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:09:48.549860 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.549836 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j"] Oct 31 00:09:48.686477 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.686440 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1376e47c-dnv8j\" (UID: \"4540c6a7-be99-4a24-afdb-1a85613bed36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:48.686629 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.686586 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llf2c\" (UniqueName: \"kubernetes.io/projected/4540c6a7-be99-4a24-afdb-1a85613bed36-kube-api-access-llf2c\") pod \"maas-api-revoke-sub-1376e47c-dnv8j\" (UID: \"4540c6a7-be99-4a24-afdb-1a85613bed36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:48.686714 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.686639 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1376e47c-dnv8j\" (UID: \"4540c6a7-be99-4a24-afdb-1a85613bed36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:48.788098 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.788010 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1376e47c-dnv8j\" (UID: \"4540c6a7-be99-4a24-afdb-1a85613bed36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:48.788098 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.788068 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llf2c\" (UniqueName: \"kubernetes.io/projected/4540c6a7-be99-4a24-afdb-1a85613bed36-kube-api-access-llf2c\") pod \"maas-api-revoke-sub-1376e47c-dnv8j\" (UID: \"4540c6a7-be99-4a24-afdb-1a85613bed36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:48.788281 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.788108 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1376e47c-dnv8j\" (UID: \"4540c6a7-be99-4a24-afdb-1a85613bed36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:48.788907 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.788884 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1376e47c-dnv8j\" (UID: \"4540c6a7-be99-4a24-afdb-1a85613bed36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:48.790689 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.790662 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1376e47c-dnv8j\" (UID: \"4540c6a7-be99-4a24-afdb-1a85613bed36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:48.801421 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.801396 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llf2c\" (UniqueName: \"kubernetes.io/projected/4540c6a7-be99-4a24-afdb-1a85613bed36-kube-api-access-llf2c\") pod \"maas-api-revoke-sub-1376e47c-dnv8j\" (UID: \"4540c6a7-be99-4a24-afdb-1a85613bed36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:48.855276 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:48.855247 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:49.136337 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:49.136312 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j"] Oct 31 00:09:49.138193 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:09:49.138162 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4540c6a7_be99_4a24_afdb_1a85613bed36.slice/crio-d70679ccd8e5a9bd7aad746f5aaa68b434d362e01fc3e74d22982a9c5a937b0f WatchSource:0}: Error finding container d70679ccd8e5a9bd7aad746f5aaa68b434d362e01fc3e74d22982a9c5a937b0f: Status 404 returned error can't find the container with id d70679ccd8e5a9bd7aad746f5aaa68b434d362e01fc3e74d22982a9c5a937b0f Oct 31 00:09:49.670666 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:49.670561 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" event={"ID":"4540c6a7-be99-4a24-afdb-1a85613bed36","Type":"ContainerStarted","Data":"f66237f71959c0002a8ba94a573420a2d77c632c671b34dfcb55d7e41c4230ef"} Oct 31 00:09:49.670666 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:49.670633 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" event={"ID":"4540c6a7-be99-4a24-afdb-1a85613bed36","Type":"ContainerStarted","Data":"d70679ccd8e5a9bd7aad746f5aaa68b434d362e01fc3e74d22982a9c5a937b0f"} Oct 31 00:09:49.691761 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:49.691704 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" podStartSLOduration=1.6916868 podStartE2EDuration="1.6916868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:49.68870128 +0000 UTC m=+3001.207702374" watchObservedRunningTime="2026-10-31 00:09:49.6916868 +0000 UTC m=+3001.210687896" Oct 31 00:09:50.678151 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:50.678120 7570 generic.go:356] "Generic (PLEG): container finished" podID="4540c6a7-be99-4a24-afdb-1a85613bed36" containerID="f66237f71959c0002a8ba94a573420a2d77c632c671b34dfcb55d7e41c4230ef" exitCode=0 Oct 31 00:09:50.678151 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:50.678156 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" event={"ID":"4540c6a7-be99-4a24-afdb-1a85613bed36","Type":"ContainerDied","Data":"f66237f71959c0002a8ba94a573420a2d77c632c671b34dfcb55d7e41c4230ef"} Oct 31 00:09:51.824869 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:51.824845 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:51.922416 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:51.922389 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-openshift-ca\") pod \"4540c6a7-be99-4a24-afdb-1a85613bed36\" (UID: \"4540c6a7-be99-4a24-afdb-1a85613bed36\") " Oct 31 00:09:51.922416 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:51.922424 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4540c6a7-be99-4a24-afdb-1a85613bed36-kube-api-access-llf2c\" (UniqueName: \"kubernetes.io/projected/4540c6a7-be99-4a24-afdb-1a85613bed36-kube-api-access-llf2c\") pod \"4540c6a7-be99-4a24-afdb-1a85613bed36\" (UID: \"4540c6a7-be99-4a24-afdb-1a85613bed36\") " Oct 31 00:09:51.922652 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:51.922444 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-xks-ca\") pod \"4540c6a7-be99-4a24-afdb-1a85613bed36\" (UID: \"4540c6a7-be99-4a24-afdb-1a85613bed36\") " Oct 31 00:09:51.922821 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:51.922797 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-openshift-ca" pod "4540c6a7-be99-4a24-afdb-1a85613bed36" (UID: "4540c6a7-be99-4a24-afdb-1a85613bed36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:51.922979 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:51.922959 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:51.924717 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:51.924691 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-xks-ca" pod "4540c6a7-be99-4a24-afdb-1a85613bed36" (UID: "4540c6a7-be99-4a24-afdb-1a85613bed36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:09:51.924827 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:51.924805 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4540c6a7-be99-4a24-afdb-1a85613bed36-kube-api-access-llf2c" pod "4540c6a7-be99-4a24-afdb-1a85613bed36" (UID: "4540c6a7-be99-4a24-afdb-1a85613bed36"). InnerVolumeSpecName "kube-api-access-llf2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:52.024334 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:52.024306 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llf2c\" (UniqueName: \"kubernetes.io/projected/4540c6a7-be99-4a24-afdb-1a85613bed36-kube-api-access-llf2c\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:52.024334 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:52.024330 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4540c6a7-be99-4a24-afdb-1a85613bed36-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:52.691285 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:52.691250 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" event={"ID":"4540c6a7-be99-4a24-afdb-1a85613bed36","Type":"ContainerDied","Data":"d70679ccd8e5a9bd7aad746f5aaa68b434d362e01fc3e74d22982a9c5a937b0f"} Oct 31 00:09:52.691285 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:52.691275 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j" Oct 31 00:09:52.691514 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:52.691335 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d70679ccd8e5a9bd7aad746f5aaa68b434d362e01fc3e74d22982a9c5a937b0f" Oct 31 00:09:58.025743 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:58.025710 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9"] Oct 31 00:09:58.028541 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:58.028513 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-95md9"] Oct 31 00:09:58.601178 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:58.601136 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7abfd60b-5291-4a6d-85e9-e7aeba227819" path="/var/lib/kubelet/pods/7abfd60b-5291-4a6d-85e9-e7aeba227819/volumes" Oct 31 00:09:59.412451 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:09:59.412412 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:09:59.620901 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.620867 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j"] Oct 31 00:09:59.621350 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.621338 7570 cpu_manager.go:414] "containerMap: removing container" podUID="4540c6a7-be99-4a24-afdb-1a85613bed36" containerName="revoke-keys" Oct 31 00:09:59.621387 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.621352 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4540c6a7-be99-4a24-afdb-1a85613bed36" containerName="revoke-keys" Oct 31 00:09:59.621488 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.621454 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4540c6a7-be99-4a24-afdb-1a85613bed36" containerName="revoke-keys" Oct 31 00:09:59.625058 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.625043 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:09:59.630335 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.630301 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:09:59.630813 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.630776 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:09:59.631603 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.631582 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:09:59.631732 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.631700 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:09:59.635685 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.635665 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j"] Oct 31 00:09:59.805377 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.805290 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltbmp\" (UniqueName: \"kubernetes.io/projected/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-kube-api-access-ltbmp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j\" (UID: \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:09:59.805565 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.805414 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j\" (UID: \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:09:59.805565 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.805498 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j\" (UID: \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:09:59.906753 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.906720 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j\" (UID: \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:09:59.906753 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.906757 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j\" (UID: \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:09:59.906939 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.906814 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltbmp\" (UniqueName: \"kubernetes.io/projected/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-kube-api-access-ltbmp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j\" (UID: \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:09:59.907494 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.907439 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j\" (UID: \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:09:59.909289 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.909267 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j\" (UID: \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:09:59.919552 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.919527 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltbmp\" (UniqueName: \"kubernetes.io/projected/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-kube-api-access-ltbmp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j\" (UID: \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:09:59.940370 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:09:59.940351 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:10:00.220523 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:00.220495 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j"] Oct 31 00:10:00.221984 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:10:00.221955 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0731f6_be26_4a8e_b3d4_2f45fb2fcc27.slice/crio-e0ff88abafb5ff631b396f7a425fd45d054f1ba9bb217c1361cdbadbde424e9b WatchSource:0}: Error finding container e0ff88abafb5ff631b396f7a425fd45d054f1ba9bb217c1361cdbadbde424e9b: Status 404 returned error can't find the container with id e0ff88abafb5ff631b396f7a425fd45d054f1ba9bb217c1361cdbadbde424e9b Oct 31 00:10:00.740431 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:00.740398 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" event={"ID":"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27","Type":"ContainerStarted","Data":"b13cab2b43cda70aefa546bff4f75c9e6c290add6144b47df16b99878029ce04"} Oct 31 00:10:00.740879 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:00.740449 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" event={"ID":"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27","Type":"ContainerStarted","Data":"e0ff88abafb5ff631b396f7a425fd45d054f1ba9bb217c1361cdbadbde424e9b"} Oct 31 00:10:00.763896 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:00.763798 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" podStartSLOduration=1.763779859 podStartE2EDuration="1.763779859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:00.762868926 +0000 UTC m=+3012.281870018" watchObservedRunningTime="2026-10-31 00:10:00.763779859 +0000 UTC m=+3012.282780949" Oct 31 00:10:01.747593 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:01.747560 7570 generic.go:356] "Generic (PLEG): container finished" podID="ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27" containerID="b13cab2b43cda70aefa546bff4f75c9e6c290add6144b47df16b99878029ce04" exitCode=0 Oct 31 00:10:01.748024 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:01.747610 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" event={"ID":"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27","Type":"ContainerDied","Data":"b13cab2b43cda70aefa546bff4f75c9e6c290add6144b47df16b99878029ce04"} Oct 31 00:10:02.888562 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:02.888538 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:10:02.939071 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:02.939041 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-xks-ca\") pod \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\" (UID: \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\") " Oct 31 00:10:02.939223 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:02.939087 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-openshift-ca\") pod \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\" (UID: \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\") " Oct 31 00:10:02.939264 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:02.939230 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-kube-api-access-ltbmp\" (UniqueName: \"kubernetes.io/projected/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-kube-api-access-ltbmp\") pod \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\" (UID: \"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27\") " Oct 31 00:10:02.939540 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:02.939518 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-openshift-ca" pod "ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27" (UID: "ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:02.939634 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:02.939621 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.941426 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:02.941401 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-kube-api-access-ltbmp" pod "ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27" (UID: "ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27"). InnerVolumeSpecName "kube-api-access-ltbmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:02.941693 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:02.941676 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-xks-ca" pod "ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27" (UID: "ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:03.041199 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:03.041124 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.041199 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:03.041152 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltbmp\" (UniqueName: \"kubernetes.io/projected/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27-kube-api-access-ltbmp\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.760779 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:03.760752 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" Oct 31 00:10:03.760947 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:03.760752 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j" event={"ID":"ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27","Type":"ContainerDied","Data":"e0ff88abafb5ff631b396f7a425fd45d054f1ba9bb217c1361cdbadbde424e9b"} Oct 31 00:10:03.760947 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:03.760878 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0ff88abafb5ff631b396f7a425fd45d054f1ba9bb217c1361cdbadbde424e9b" Oct 31 00:10:09.751036 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:09.751004 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk"] Oct 31 00:10:09.783721 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:09.783690 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb"] Oct 31 00:10:10.806937 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:10.806895 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j"] Oct 31 00:10:10.809671 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:10.809650 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93f1ad-pvn6j"] Oct 31 00:10:12.601281 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:12.601250 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27" path="/var/lib/kubelet/pods/ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27/volumes" Oct 31 00:10:14.751878 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:14.751838 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" podUID="a782d992-67cc-4c0a-ad76-5652e94178b1" containerName="payload-processing" containerID="cri-o://ea8d5ac2543b19c9c5b5b46982cd94b2ead83dff61b52791f2b3c6c69eaee592" gracePeriod=25 Oct 31 00:10:14.784167 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:14.784133 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" podUID="c0eb9a33-1845-4dee-aab3-0bc0ed460447" containerName="payload-pre-processing" containerID="cri-o://0a8cead280b2f7d0e6f907767e70037c3b6889a07544f6cd8ee3f95ee1adc394" gracePeriod=25 Oct 31 00:10:17.027884 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:17.027841 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb"] Oct 31 00:10:17.030558 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:17.030540 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l6jxb"] Oct 31 00:10:18.073590 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:18.073556 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9"] Oct 31 00:10:18.074009 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:18.073828 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" containerID="cri-o://baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9" gracePeriod=30 Oct 31 00:10:18.601048 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:18.601014 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a79d5c6-7859-4f4c-a8d4-5713eb8185ac" path="/var/lib/kubelet/pods/5a79d5c6-7859-4f4c-a8d4-5713eb8185ac/volumes" Oct 31 00:10:19.357130 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.357097 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5"] Oct 31 00:10:19.363189 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.363164 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-swtz5"] Oct 31 00:10:19.383614 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.383590 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx"] Oct 31 00:10:19.384050 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.384037 7570 cpu_manager.go:414] "containerMap: removing container" podUID="ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27" containerName="revoke-keys" Oct 31 00:10:19.384101 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.384052 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27" containerName="revoke-keys" Oct 31 00:10:19.384178 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.384169 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0731f6-be26-4a8e-b3d4-2f45fb2fcc27" containerName="revoke-keys" Oct 31 00:10:19.387560 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.387546 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:19.393409 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.393346 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:10:19.395004 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.394542 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:10:19.395004 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.394574 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:10:19.395004 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.394581 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:10:19.397699 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.397675 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx"] Oct 31 00:10:19.512225 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.512191 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bftqx\" (UID: \"52006e8e-96f6-4cf0-8654-744bea2151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:19.512377 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.512261 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bftqx\" (UID: \"52006e8e-96f6-4cf0-8654-744bea2151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:19.512419 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.512406 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n497\" (UniqueName: \"kubernetes.io/projected/52006e8e-96f6-4cf0-8654-744bea2151d2-kube-api-access-2n497\") pod \"maas-api-revoke-sub-3d4156f1-bftqx\" (UID: \"52006e8e-96f6-4cf0-8654-744bea2151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:19.613979 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.613903 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bftqx\" (UID: \"52006e8e-96f6-4cf0-8654-744bea2151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:19.613979 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.613943 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bftqx\" (UID: \"52006e8e-96f6-4cf0-8654-744bea2151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:19.614152 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.614022 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n497\" (UniqueName: \"kubernetes.io/projected/52006e8e-96f6-4cf0-8654-744bea2151d2-kube-api-access-2n497\") pod \"maas-api-revoke-sub-3d4156f1-bftqx\" (UID: \"52006e8e-96f6-4cf0-8654-744bea2151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:19.614601 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.614582 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bftqx\" (UID: \"52006e8e-96f6-4cf0-8654-744bea2151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:19.616482 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.616450 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bftqx\" (UID: \"52006e8e-96f6-4cf0-8654-744bea2151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:19.626632 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.626607 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n497\" (UniqueName: \"kubernetes.io/projected/52006e8e-96f6-4cf0-8654-744bea2151d2-kube-api-access-2n497\") pod \"maas-api-revoke-sub-3d4156f1-bftqx\" (UID: \"52006e8e-96f6-4cf0-8654-744bea2151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:19.703570 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.703535 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:19.985896 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:19.985873 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx"] Oct 31 00:10:19.988091 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:10:19.988061 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52006e8e_96f6_4cf0_8654_744bea2151d2.slice/crio-2ef5a1a846ddcccbce82d6cec8c6fd03288fbcea08bd17cb0f43625bbbed7f73 WatchSource:0}: Error finding container 2ef5a1a846ddcccbce82d6cec8c6fd03288fbcea08bd17cb0f43625bbbed7f73: Status 404 returned error can't find the container with id 2ef5a1a846ddcccbce82d6cec8c6fd03288fbcea08bd17cb0f43625bbbed7f73 Oct 31 00:10:20.367131 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:20.367093 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" podUID="c0eb9a33-1845-4dee-aab3-0bc0ed460447" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:10:20.367131 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:20.367117 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" podUID="a782d992-67cc-4c0a-ad76-5652e94178b1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:10:20.602104 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:20.602075 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809bb7bf-e365-4014-a8dc-d05dd97fc155" path="/var/lib/kubelet/pods/809bb7bf-e365-4014-a8dc-d05dd97fc155/volumes" Oct 31 00:10:20.862204 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:20.862127 7570 generic.go:356] "Generic (PLEG): container finished" podID="52006e8e-96f6-4cf0-8654-744bea2151d2" containerID="298941680c17d5c656420abb128b82d45868dd8617579f6851dfff31d8ae7946" exitCode=0 Oct 31 00:10:20.862356 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:20.862217 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" event={"ID":"52006e8e-96f6-4cf0-8654-744bea2151d2","Type":"ContainerDied","Data":"298941680c17d5c656420abb128b82d45868dd8617579f6851dfff31d8ae7946"} Oct 31 00:10:20.862356 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:20.862279 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" event={"ID":"52006e8e-96f6-4cf0-8654-744bea2151d2","Type":"ContainerStarted","Data":"2ef5a1a846ddcccbce82d6cec8c6fd03288fbcea08bd17cb0f43625bbbed7f73"} Oct 31 00:10:22.010481 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.010428 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:22.138370 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.138277 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-xks-ca\") pod \"52006e8e-96f6-4cf0-8654-744bea2151d2\" (UID: \"52006e8e-96f6-4cf0-8654-744bea2151d2\") " Oct 31 00:10:22.138370 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.138328 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-openshift-ca\") pod \"52006e8e-96f6-4cf0-8654-744bea2151d2\" (UID: \"52006e8e-96f6-4cf0-8654-744bea2151d2\") " Oct 31 00:10:22.138610 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.138390 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52006e8e-96f6-4cf0-8654-744bea2151d2-kube-api-access-2n497\" (UniqueName: \"kubernetes.io/projected/52006e8e-96f6-4cf0-8654-744bea2151d2-kube-api-access-2n497\") pod \"52006e8e-96f6-4cf0-8654-744bea2151d2\" (UID: \"52006e8e-96f6-4cf0-8654-744bea2151d2\") " Oct 31 00:10:22.138745 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.138723 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-openshift-ca" pod "52006e8e-96f6-4cf0-8654-744bea2151d2" (UID: "52006e8e-96f6-4cf0-8654-744bea2151d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:22.140622 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.140599 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-xks-ca" pod "52006e8e-96f6-4cf0-8654-744bea2151d2" (UID: "52006e8e-96f6-4cf0-8654-744bea2151d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:22.140737 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.140645 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52006e8e-96f6-4cf0-8654-744bea2151d2-kube-api-access-2n497" pod "52006e8e-96f6-4cf0-8654-744bea2151d2" (UID: "52006e8e-96f6-4cf0-8654-744bea2151d2"). InnerVolumeSpecName "kube-api-access-2n497". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:22.239442 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.239415 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n497\" (UniqueName: \"kubernetes.io/projected/52006e8e-96f6-4cf0-8654-744bea2151d2-kube-api-access-2n497\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:22.239442 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.239439 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:22.239646 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.239449 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52006e8e-96f6-4cf0-8654-744bea2151d2-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:22.875817 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.875738 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" Oct 31 00:10:22.875973 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.875740 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx" event={"ID":"52006e8e-96f6-4cf0-8654-744bea2151d2","Type":"ContainerDied","Data":"2ef5a1a846ddcccbce82d6cec8c6fd03288fbcea08bd17cb0f43625bbbed7f73"} Oct 31 00:10:22.875973 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:22.875866 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ef5a1a846ddcccbce82d6cec8c6fd03288fbcea08bd17cb0f43625bbbed7f73" Oct 31 00:10:23.327217 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.327194 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:10:23.455005 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.454933 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-kube-api-access-cgvqs\" (UniqueName: \"kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-kube-api-access-cgvqs\") pod \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " Oct 31 00:10:23.455005 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.454982 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-envoy\") pod \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " Oct 31 00:10:23.455173 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.455008 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-data\") pod \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " Oct 31 00:10:23.455173 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.455110 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-token\" (UniqueName: \"kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-token\") pod \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " Oct 31 00:10:23.455173 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.455148 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istiod-ca-cert\") pod \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " Oct 31 00:10:23.455351 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.455215 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-podinfo\") pod \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " Oct 31 00:10:23.455351 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.455280 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-data" pod "57ae1ff0-9b19-456c-b564-8e21c1b7baf5" (UID: "57ae1ff0-9b19-456c-b564-8e21c1b7baf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:23.455351 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.455300 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-credential-socket\") pod \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " Oct 31 00:10:23.455351 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.455339 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-certs\") pod \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " Oct 31 00:10:23.455619 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.455416 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-socket\") pod \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\" (UID: \"57ae1ff0-9b19-456c-b564-8e21c1b7baf5\") " Oct 31 00:10:23.455679 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.455629 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-credential-socket" pod "57ae1ff0-9b19-456c-b564-8e21c1b7baf5" (UID: "57ae1ff0-9b19-456c-b564-8e21c1b7baf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:23.455679 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.455641 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istiod-ca-cert" pod "57ae1ff0-9b19-456c-b564-8e21c1b7baf5" (UID: "57ae1ff0-9b19-456c-b564-8e21c1b7baf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:23.455886 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.455838 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-socket" pod "57ae1ff0-9b19-456c-b564-8e21c1b7baf5" (UID: "57ae1ff0-9b19-456c-b564-8e21c1b7baf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:23.456001 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.455907 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-certs" pod "57ae1ff0-9b19-456c-b564-8e21c1b7baf5" (UID: "57ae1ff0-9b19-456c-b564-8e21c1b7baf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:23.456173 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.456157 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:23.456239 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.456178 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:23.456239 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.456194 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:23.456239 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.456208 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:23.456239 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.456224 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:23.457415 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.457395 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-kube-api-access-cgvqs" pod "57ae1ff0-9b19-456c-b564-8e21c1b7baf5" (UID: "57ae1ff0-9b19-456c-b564-8e21c1b7baf5"). InnerVolumeSpecName "kube-api-access-cgvqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:23.457539 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.457506 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-envoy" pod "57ae1ff0-9b19-456c-b564-8e21c1b7baf5" (UID: "57ae1ff0-9b19-456c-b564-8e21c1b7baf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:23.458010 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.457993 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-podinfo" pod "57ae1ff0-9b19-456c-b564-8e21c1b7baf5" (UID: "57ae1ff0-9b19-456c-b564-8e21c1b7baf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:10:23.458053 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.458017 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-token" pod "57ae1ff0-9b19-456c-b564-8e21c1b7baf5" (UID: "57ae1ff0-9b19-456c-b564-8e21c1b7baf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:23.557450 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.557407 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:23.557450 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.557439 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgvqs\" (UniqueName: \"kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-kube-api-access-cgvqs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:23.557450 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.557449 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:23.557450 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.557473 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ae1ff0-9b19-456c-b564-8e21c1b7baf5-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:23.883041 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.883012 7570 generic.go:356] "Generic (PLEG): container finished" podID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerID="baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9" exitCode=0 Oct 31 00:10:23.883215 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.883064 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" Oct 31 00:10:23.883215 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.883102 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" event={"ID":"57ae1ff0-9b19-456c-b564-8e21c1b7baf5","Type":"ContainerDied","Data":"baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9"} Oct 31 00:10:23.883215 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.883159 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9" event={"ID":"57ae1ff0-9b19-456c-b564-8e21c1b7baf5","Type":"ContainerDied","Data":"24a0a24d3e6769dd70d8682a8ddd4c3d52af55433e06470713c8b89df626248e"} Oct 31 00:10:23.883215 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.883182 7570 scope.go:122] "RemoveContainer" containerID="baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9" Oct 31 00:10:23.893659 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.893642 7570 scope.go:122] "RemoveContainer" containerID="be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe" Oct 31 00:10:23.902926 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.902904 7570 scope.go:122] "RemoveContainer" containerID="baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9" Oct 31 00:10:23.903208 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:10:23.903190 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9\": container with ID starting with baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9 not found: ID does not exist" containerID="baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9" Oct 31 00:10:23.903256 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.903218 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9"} err="failed to get container status \"baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9\": rpc error: code = NotFound desc = could not find container \"baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9\": container with ID starting with baff66552cca4dfa6372b60399f80833df8c42496bbe230e75a4ef54556f8fe9 not found: ID does not exist" Oct 31 00:10:23.903256 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.903235 7570 scope.go:122] "RemoveContainer" containerID="be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe" Oct 31 00:10:23.903494 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:10:23.903454 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe\": container with ID starting with be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe not found: ID does not exist" containerID="be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe" Oct 31 00:10:23.903564 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.903503 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe"} err="failed to get container status \"be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe\": rpc error: code = NotFound desc = could not find container \"be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe\": container with ID starting with be5eccc3b3d9f5b3c84313cc5689bd0b6b62fe79cf6d9c8e50d9db9680527bbe not found: ID does not exist" Oct 31 00:10:23.907747 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.907729 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9"] Oct 31 00:10:23.911926 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:23.911906 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-93f1ad-openshift-default-575b74c6fcz7vh9"] Oct 31 00:10:24.601165 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:24.601132 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" path="/var/lib/kubelet/pods/57ae1ff0-9b19-456c-b564-8e21c1b7baf5/volumes" Oct 31 00:10:28.024576 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:28.024526 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z"] Oct 31 00:10:28.027655 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:28.027630 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9pc8z"] Oct 31 00:10:28.601174 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:28.601143 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63eddb85-3779-4cf7-8a98-2e51939e7639" path="/var/lib/kubelet/pods/63eddb85-3779-4cf7-8a98-2e51939e7639/volumes" Oct 31 00:10:29.411310 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:10:29.411268 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:10:30.366740 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:30.366704 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" podUID="a782d992-67cc-4c0a-ad76-5652e94178b1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:10:30.366890 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:30.366776 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" podUID="c0eb9a33-1845-4dee-aab3-0bc0ed460447" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:10:34.954170 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:34.954124 7570 generic.go:356] "Generic (PLEG): container finished" podID="c0eb9a33-1845-4dee-aab3-0bc0ed460447" containerID="0a8cead280b2f7d0e6f907767e70037c3b6889a07544f6cd8ee3f95ee1adc394" exitCode=0 Oct 31 00:10:34.954736 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:34.954245 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" event={"ID":"c0eb9a33-1845-4dee-aab3-0bc0ed460447","Type":"ContainerDied","Data":"0a8cead280b2f7d0e6f907767e70037c3b6889a07544f6cd8ee3f95ee1adc394"} Oct 31 00:10:34.965155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:34.965116 7570 generic.go:356] "Generic (PLEG): container finished" podID="a782d992-67cc-4c0a-ad76-5652e94178b1" containerID="ea8d5ac2543b19c9c5b5b46982cd94b2ead83dff61b52791f2b3c6c69eaee592" exitCode=0 Oct 31 00:10:34.965286 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:34.965231 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" event={"ID":"a782d992-67cc-4c0a-ad76-5652e94178b1","Type":"ContainerDied","Data":"ea8d5ac2543b19c9c5b5b46982cd94b2ead83dff61b52791f2b3c6c69eaee592"} Oct 31 00:10:35.040812 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.040788 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" Oct 31 00:10:35.071222 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.071199 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a782d992-67cc-4c0a-ad76-5652e94178b1-kube-api-access-z79k9\" (UniqueName: \"kubernetes.io/projected/a782d992-67cc-4c0a-ad76-5652e94178b1-kube-api-access-z79k9\") pod \"a782d992-67cc-4c0a-ad76-5652e94178b1\" (UID: \"a782d992-67cc-4c0a-ad76-5652e94178b1\") " Oct 31 00:10:35.071350 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.071255 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a782d992-67cc-4c0a-ad76-5652e94178b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a782d992-67cc-4c0a-ad76-5652e94178b1-plugins-config-volume\") pod \"a782d992-67cc-4c0a-ad76-5652e94178b1\" (UID: \"a782d992-67cc-4c0a-ad76-5652e94178b1\") " Oct 31 00:10:35.071724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.071701 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a782d992-67cc-4c0a-ad76-5652e94178b1-plugins-config-volume" pod "a782d992-67cc-4c0a-ad76-5652e94178b1" (UID: "a782d992-67cc-4c0a-ad76-5652e94178b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:35.073583 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.073557 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a782d992-67cc-4c0a-ad76-5652e94178b1-kube-api-access-z79k9" pod "a782d992-67cc-4c0a-ad76-5652e94178b1" (UID: "a782d992-67cc-4c0a-ad76-5652e94178b1"). InnerVolumeSpecName "kube-api-access-z79k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:35.077522 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.077503 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" Oct 31 00:10:35.172676 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.172647 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0eb9a33-1845-4dee-aab3-0bc0ed460447-kube-api-access-m9zxt\" (UniqueName: \"kubernetes.io/projected/c0eb9a33-1845-4dee-aab3-0bc0ed460447-kube-api-access-m9zxt\") pod \"c0eb9a33-1845-4dee-aab3-0bc0ed460447\" (UID: \"c0eb9a33-1845-4dee-aab3-0bc0ed460447\") " Oct 31 00:10:35.172830 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.172719 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0eb9a33-1845-4dee-aab3-0bc0ed460447-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0eb9a33-1845-4dee-aab3-0bc0ed460447-plugins-config-volume\") pod \"c0eb9a33-1845-4dee-aab3-0bc0ed460447\" (UID: \"c0eb9a33-1845-4dee-aab3-0bc0ed460447\") " Oct 31 00:10:35.172953 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.172940 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z79k9\" (UniqueName: \"kubernetes.io/projected/a782d992-67cc-4c0a-ad76-5652e94178b1-kube-api-access-z79k9\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:35.172994 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.172956 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a782d992-67cc-4c0a-ad76-5652e94178b1-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:35.173088 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.173069 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0eb9a33-1845-4dee-aab3-0bc0ed460447-plugins-config-volume" pod "c0eb9a33-1845-4dee-aab3-0bc0ed460447" (UID: "c0eb9a33-1845-4dee-aab3-0bc0ed460447"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:35.174990 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.174970 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0eb9a33-1845-4dee-aab3-0bc0ed460447-kube-api-access-m9zxt" pod "c0eb9a33-1845-4dee-aab3-0bc0ed460447" (UID: "c0eb9a33-1845-4dee-aab3-0bc0ed460447"). InnerVolumeSpecName "kube-api-access-m9zxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:35.274279 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.274200 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9zxt\" (UniqueName: \"kubernetes.io/projected/c0eb9a33-1845-4dee-aab3-0bc0ed460447-kube-api-access-m9zxt\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:35.274279 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.274233 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0eb9a33-1845-4dee-aab3-0bc0ed460447-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:35.972398 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.972371 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" Oct 31 00:10:35.972398 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.972389 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb" event={"ID":"c0eb9a33-1845-4dee-aab3-0bc0ed460447","Type":"ContainerDied","Data":"6d1bd8d85f64b80963810ae2b5515f0c0664254fe14267db5dd9677ddc5b3220"} Oct 31 00:10:35.972963 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.972480 7570 scope.go:122] "RemoveContainer" containerID="0a8cead280b2f7d0e6f907767e70037c3b6889a07544f6cd8ee3f95ee1adc394" Oct 31 00:10:35.974088 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.974067 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" event={"ID":"a782d992-67cc-4c0a-ad76-5652e94178b1","Type":"ContainerDied","Data":"e861ffea0b209d244f1630dbb9f0d54cffc5beddb7ee4b8872419040c080f6f9"} Oct 31 00:10:35.974235 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.974217 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk" Oct 31 00:10:35.988656 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.988637 7570 scope.go:122] "RemoveContainer" containerID="ea8d5ac2543b19c9c5b5b46982cd94b2ead83dff61b52791f2b3c6c69eaee592" Oct 31 00:10:35.999056 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:35.999035 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb"] Oct 31 00:10:36.004840 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:36.004816 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-93f1ad-5c4d76zxdb"] Oct 31 00:10:36.016765 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:36.016744 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk"] Oct 31 00:10:36.022507 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:36.022426 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-93f1ad-7b5f895f6ggzjk"] Oct 31 00:10:36.601203 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:36.601171 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a782d992-67cc-4c0a-ad76-5652e94178b1" path="/var/lib/kubelet/pods/a782d992-67cc-4c0a-ad76-5652e94178b1/volumes" Oct 31 00:10:36.601573 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:36.601557 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0eb9a33-1845-4dee-aab3-0bc0ed460447" path="/var/lib/kubelet/pods/c0eb9a33-1845-4dee-aab3-0bc0ed460447/volumes" Oct 31 00:10:44.324588 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:44.324556 7570 scope.go:122] "RemoveContainer" containerID="0ed8ee7940f11553dca12bc017f6736a26015c04176ca5f929ee75f2244092a0" Oct 31 00:10:55.034486 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:55.034428 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv"] Oct 31 00:10:55.036428 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:55.036408 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-txkxv"] Oct 31 00:10:56.601298 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:10:56.601265 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252ce7ac-016d-4803-af32-b0812df8d361" path="/var/lib/kubelet/pods/252ce7ac-016d-4803-af32-b0812df8d361/volumes" Oct 31 00:10:59.411101 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:10:59.411063 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:11:05.029150 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:05.029113 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz"] Oct 31 00:11:05.032313 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:05.032290 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5hkz"] Oct 31 00:11:06.608913 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:06.608272 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8065de8-6c7b-4b7b-a044-0cebb0f930a7" path="/var/lib/kubelet/pods/f8065de8-6c7b-4b7b-a044-0cebb0f930a7/volumes" Oct 31 00:11:26.034785 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:26.034746 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6"] Oct 31 00:11:26.037269 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:26.037241 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sgtq6"] Oct 31 00:11:26.600896 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:26.600862 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76dc2375-44d4-42ab-b717-7bafbce8eb97" path="/var/lib/kubelet/pods/76dc2375-44d4-42ab-b717-7bafbce8eb97/volumes" Oct 31 00:11:29.411907 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:11:29.411872 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:11:44.024425 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:44.024387 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7"] Oct 31 00:11:44.028375 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:44.028347 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8s7b7"] Oct 31 00:11:44.362613 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:44.362573 7570 scope.go:122] "RemoveContainer" containerID="16c3f4f57408294dae89f5e3e188ffe951047b5e1d15ff4626926e622be7ab91" Oct 31 00:11:44.372640 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:44.372620 7570 scope.go:122] "RemoveContainer" containerID="9510c9ff322634d2aadb923a23f0689fda9cb643ac41059d56abf72792fb65a0" Oct 31 00:11:44.381282 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:44.381260 7570 scope.go:122] "RemoveContainer" containerID="2b4189433db6c405d30bf499a00e39c25d372dba454dcc31c0818aa2af8d554a" Oct 31 00:11:44.601169 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:44.601134 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714f825a-d72f-4cc6-b14a-4b71590b2ce3" path="/var/lib/kubelet/pods/714f825a-d72f-4cc6-b14a-4b71590b2ce3/volumes" Oct 31 00:11:58.176649 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.176613 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274"] Oct 31 00:11:58.177064 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177034 7570 cpu_manager.go:414] "containerMap: removing container" podUID="c0eb9a33-1845-4dee-aab3-0bc0ed460447" containerName="payload-pre-processing" Oct 31 00:11:58.177064 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177046 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0eb9a33-1845-4dee-aab3-0bc0ed460447" containerName="payload-pre-processing" Oct 31 00:11:58.177064 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177064 7570 cpu_manager.go:414] "containerMap: removing container" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" Oct 31 00:11:58.177167 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177070 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" Oct 31 00:11:58.177167 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177087 7570 cpu_manager.go:414] "containerMap: removing container" podUID="52006e8e-96f6-4cf0-8654-744bea2151d2" containerName="revoke-keys" Oct 31 00:11:58.177167 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177091 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52006e8e-96f6-4cf0-8654-744bea2151d2" containerName="revoke-keys" Oct 31 00:11:58.177167 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177104 7570 cpu_manager.go:414] "containerMap: removing container" podUID="a782d992-67cc-4c0a-ad76-5652e94178b1" containerName="payload-processing" Oct 31 00:11:58.177167 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177109 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a782d992-67cc-4c0a-ad76-5652e94178b1" containerName="payload-processing" Oct 31 00:11:58.177167 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177123 7570 cpu_manager.go:414] "containerMap: removing container" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" Oct 31 00:11:58.177167 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177127 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" Oct 31 00:11:58.177364 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177186 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" Oct 31 00:11:58.177364 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177196 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="52006e8e-96f6-4cf0-8654-744bea2151d2" containerName="revoke-keys" Oct 31 00:11:58.177364 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177204 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a782d992-67cc-4c0a-ad76-5652e94178b1" containerName="payload-processing" Oct 31 00:11:58.177364 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177212 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ae1ff0-9b19-456c-b564-8e21c1b7baf5" containerName="istio-proxy" Oct 31 00:11:58.177364 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.177220 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0eb9a33-1845-4dee-aab3-0bc0ed460447" containerName="payload-pre-processing" Oct 31 00:11:58.180492 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.180451 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:11:58.188356 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.188334 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:11:58.188622 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.188605 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:11:58.188760 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.188405 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:11:58.188760 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.188444 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:11:58.190683 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.190664 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274"] Oct 31 00:11:58.281127 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.281095 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-4f274\" (UID: \"49f346c4-b2c1-433a-82f3-d9de686c6e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:11:58.281127 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.281129 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-4f274\" (UID: \"49f346c4-b2c1-433a-82f3-d9de686c6e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:11:58.281312 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.281146 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pplzz\" (UniqueName: \"kubernetes.io/projected/49f346c4-b2c1-433a-82f3-d9de686c6e40-kube-api-access-pplzz\") pod \"maas-api-revoke-sub-d576e68b-4f274\" (UID: \"49f346c4-b2c1-433a-82f3-d9de686c6e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:11:58.382422 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.382390 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-4f274\" (UID: \"49f346c4-b2c1-433a-82f3-d9de686c6e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:11:58.382422 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.382423 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-4f274\" (UID: \"49f346c4-b2c1-433a-82f3-d9de686c6e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:11:58.382665 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.382442 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pplzz\" (UniqueName: \"kubernetes.io/projected/49f346c4-b2c1-433a-82f3-d9de686c6e40-kube-api-access-pplzz\") pod \"maas-api-revoke-sub-d576e68b-4f274\" (UID: \"49f346c4-b2c1-433a-82f3-d9de686c6e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:11:58.383050 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.383026 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-4f274\" (UID: \"49f346c4-b2c1-433a-82f3-d9de686c6e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:11:58.385055 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.385034 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-4f274\" (UID: \"49f346c4-b2c1-433a-82f3-d9de686c6e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:11:58.395440 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.395418 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pplzz\" (UniqueName: \"kubernetes.io/projected/49f346c4-b2c1-433a-82f3-d9de686c6e40-kube-api-access-pplzz\") pod \"maas-api-revoke-sub-d576e68b-4f274\" (UID: \"49f346c4-b2c1-433a-82f3-d9de686c6e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:11:58.495854 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.495793 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:11:58.790012 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:58.789949 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274"] Oct 31 00:11:58.792533 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:11:58.792503 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f346c4_b2c1_433a_82f3_d9de686c6e40.slice/crio-3074a79df1555f9465bf549d2876da531abd30c002ffa636269f3481a09e5f05 WatchSource:0}: Error finding container 3074a79df1555f9465bf549d2876da531abd30c002ffa636269f3481a09e5f05: Status 404 returned error can't find the container with id 3074a79df1555f9465bf549d2876da531abd30c002ffa636269f3481a09e5f05 Oct 31 00:11:59.412158 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:11:59.412070 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:11:59.473359 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:59.473320 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" event={"ID":"49f346c4-b2c1-433a-82f3-d9de686c6e40","Type":"ContainerStarted","Data":"8d0b7303c7f92eed8b63eb107a2b72d08c1ca91240b225e7c186e55ef1fdac4a"} Oct 31 00:11:59.473537 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:11:59.473380 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" event={"ID":"49f346c4-b2c1-433a-82f3-d9de686c6e40","Type":"ContainerStarted","Data":"3074a79df1555f9465bf549d2876da531abd30c002ffa636269f3481a09e5f05"} Oct 31 00:12:00.481187 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:00.481158 7570 generic.go:356] "Generic (PLEG): container finished" podID="49f346c4-b2c1-433a-82f3-d9de686c6e40" containerID="8d0b7303c7f92eed8b63eb107a2b72d08c1ca91240b225e7c186e55ef1fdac4a" exitCode=0 Oct 31 00:12:00.481602 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:00.481249 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" event={"ID":"49f346c4-b2c1-433a-82f3-d9de686c6e40","Type":"ContainerDied","Data":"8d0b7303c7f92eed8b63eb107a2b72d08c1ca91240b225e7c186e55ef1fdac4a"} Oct 31 00:12:01.628133 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:01.628105 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:12:01.712451 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:01.712421 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-xks-ca\") pod \"49f346c4-b2c1-433a-82f3-d9de686c6e40\" (UID: \"49f346c4-b2c1-433a-82f3-d9de686c6e40\") " Oct 31 00:12:01.712451 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:01.712476 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-openshift-ca\") pod \"49f346c4-b2c1-433a-82f3-d9de686c6e40\" (UID: \"49f346c4-b2c1-433a-82f3-d9de686c6e40\") " Oct 31 00:12:01.712670 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:01.712513 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f346c4-b2c1-433a-82f3-d9de686c6e40-kube-api-access-pplzz\" (UniqueName: \"kubernetes.io/projected/49f346c4-b2c1-433a-82f3-d9de686c6e40-kube-api-access-pplzz\") pod \"49f346c4-b2c1-433a-82f3-d9de686c6e40\" (UID: \"49f346c4-b2c1-433a-82f3-d9de686c6e40\") " Oct 31 00:12:01.712892 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:01.712860 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-openshift-ca" pod "49f346c4-b2c1-433a-82f3-d9de686c6e40" (UID: "49f346c4-b2c1-433a-82f3-d9de686c6e40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:01.714842 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:01.714821 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-xks-ca" pod "49f346c4-b2c1-433a-82f3-d9de686c6e40" (UID: "49f346c4-b2c1-433a-82f3-d9de686c6e40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:01.714936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:01.714917 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f346c4-b2c1-433a-82f3-d9de686c6e40-kube-api-access-pplzz" pod "49f346c4-b2c1-433a-82f3-d9de686c6e40" (UID: "49f346c4-b2c1-433a-82f3-d9de686c6e40"). InnerVolumeSpecName "kube-api-access-pplzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:01.814035 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:01.813975 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:01.814035 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:01.814001 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f346c4-b2c1-433a-82f3-d9de686c6e40-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:01.814035 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:01.814011 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pplzz\" (UniqueName: \"kubernetes.io/projected/49f346c4-b2c1-433a-82f3-d9de686c6e40-kube-api-access-pplzz\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:02.494668 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:02.494640 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" Oct 31 00:12:02.494668 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:02.494655 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274" event={"ID":"49f346c4-b2c1-433a-82f3-d9de686c6e40","Type":"ContainerDied","Data":"3074a79df1555f9465bf549d2876da531abd30c002ffa636269f3481a09e5f05"} Oct 31 00:12:02.494869 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:02.494699 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3074a79df1555f9465bf549d2876da531abd30c002ffa636269f3481a09e5f05" Oct 31 00:12:03.022658 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:03.022574 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq"] Oct 31 00:12:03.025950 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:03.025920 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2tqwq"] Oct 31 00:12:04.600873 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:04.600840 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b4f6b9-edb6-4f70-b108-087dec2e2093" path="/var/lib/kubelet/pods/e3b4f6b9-edb6-4f70-b108-087dec2e2093/volumes" Oct 31 00:12:17.254636 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.254583 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk"] Oct 31 00:12:17.255234 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.255213 7570 cpu_manager.go:414] "containerMap: removing container" podUID="49f346c4-b2c1-433a-82f3-d9de686c6e40" containerName="revoke-keys" Oct 31 00:12:17.255310 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.255232 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f346c4-b2c1-433a-82f3-d9de686c6e40" containerName="revoke-keys" Oct 31 00:12:17.255442 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.255427 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f346c4-b2c1-433a-82f3-d9de686c6e40" containerName="revoke-keys" Oct 31 00:12:17.258980 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.258960 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:17.264138 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.264116 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:12:17.265155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.264376 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:12:17.265155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.264448 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:12:17.265155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.264507 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:12:17.266890 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.266868 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk"] Oct 31 00:12:17.396391 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.396353 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jdjqk\" (UID: \"d971e795-94ee-4fbc-809a-9d605c2e7b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:17.396576 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.396522 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jdjqk\" (UID: \"d971e795-94ee-4fbc-809a-9d605c2e7b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:17.396644 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.396629 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzv5z\" (UniqueName: \"kubernetes.io/projected/d971e795-94ee-4fbc-809a-9d605c2e7b29-kube-api-access-fzv5z\") pod \"maas-api-revoke-sub-d2bc93bc-jdjqk\" (UID: \"d971e795-94ee-4fbc-809a-9d605c2e7b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:17.497877 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.497843 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jdjqk\" (UID: \"d971e795-94ee-4fbc-809a-9d605c2e7b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:17.498055 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.497934 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzv5z\" (UniqueName: \"kubernetes.io/projected/d971e795-94ee-4fbc-809a-9d605c2e7b29-kube-api-access-fzv5z\") pod \"maas-api-revoke-sub-d2bc93bc-jdjqk\" (UID: \"d971e795-94ee-4fbc-809a-9d605c2e7b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:17.498055 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.497964 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jdjqk\" (UID: \"d971e795-94ee-4fbc-809a-9d605c2e7b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:17.498645 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.498625 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jdjqk\" (UID: \"d971e795-94ee-4fbc-809a-9d605c2e7b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:17.500504 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.500480 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jdjqk\" (UID: \"d971e795-94ee-4fbc-809a-9d605c2e7b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:17.510648 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.510587 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzv5z\" (UniqueName: \"kubernetes.io/projected/d971e795-94ee-4fbc-809a-9d605c2e7b29-kube-api-access-fzv5z\") pod \"maas-api-revoke-sub-d2bc93bc-jdjqk\" (UID: \"d971e795-94ee-4fbc-809a-9d605c2e7b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:17.575897 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.575871 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:17.857047 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:17.857020 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk"] Oct 31 00:12:17.858737 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:12:17.858702 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd971e795_94ee_4fbc_809a_9d605c2e7b29.slice/crio-0eb9e7a828795415cfe658b0089b1ecd9d3d7ea9f3ea8c2907c8512434fafead WatchSource:0}: Error finding container 0eb9e7a828795415cfe658b0089b1ecd9d3d7ea9f3ea8c2907c8512434fafead: Status 404 returned error can't find the container with id 0eb9e7a828795415cfe658b0089b1ecd9d3d7ea9f3ea8c2907c8512434fafead Oct 31 00:12:18.603165 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:18.603129 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" event={"ID":"d971e795-94ee-4fbc-809a-9d605c2e7b29","Type":"ContainerStarted","Data":"bf1fe8d0dd76fbdff824c2f1ee0908cbaa5b29068eda0fe609b37543721b09a9"} Oct 31 00:12:18.603574 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:18.603182 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" event={"ID":"d971e795-94ee-4fbc-809a-9d605c2e7b29","Type":"ContainerStarted","Data":"0eb9e7a828795415cfe658b0089b1ecd9d3d7ea9f3ea8c2907c8512434fafead"} Oct 31 00:12:18.622584 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:18.622538 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" podStartSLOduration=1.622525234 podStartE2EDuration="1.622525234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:18.622092949 +0000 UTC m=+3150.141094038" watchObservedRunningTime="2026-10-31 00:12:18.622525234 +0000 UTC m=+3150.141526324" Oct 31 00:12:19.611843 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:19.611799 7570 generic.go:356] "Generic (PLEG): container finished" podID="d971e795-94ee-4fbc-809a-9d605c2e7b29" containerID="bf1fe8d0dd76fbdff824c2f1ee0908cbaa5b29068eda0fe609b37543721b09a9" exitCode=0 Oct 31 00:12:19.612311 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:19.611885 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" event={"ID":"d971e795-94ee-4fbc-809a-9d605c2e7b29","Type":"ContainerDied","Data":"bf1fe8d0dd76fbdff824c2f1ee0908cbaa5b29068eda0fe609b37543721b09a9"} Oct 31 00:12:20.034525 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:20.034405 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn"] Oct 31 00:12:20.036896 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:20.036873 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6v7sn"] Oct 31 00:12:20.600620 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:20.600589 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ef3464-bc1e-4efa-b829-7264ef415ab7" path="/var/lib/kubelet/pods/60ef3464-bc1e-4efa-b829-7264ef415ab7/volumes" Oct 31 00:12:20.752062 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:20.752039 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:20.934972 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:20.934891 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-xks-ca\") pod \"d971e795-94ee-4fbc-809a-9d605c2e7b29\" (UID: \"d971e795-94ee-4fbc-809a-9d605c2e7b29\") " Oct 31 00:12:20.934972 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:20.934959 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d971e795-94ee-4fbc-809a-9d605c2e7b29-kube-api-access-fzv5z\" (UniqueName: \"kubernetes.io/projected/d971e795-94ee-4fbc-809a-9d605c2e7b29-kube-api-access-fzv5z\") pod \"d971e795-94ee-4fbc-809a-9d605c2e7b29\" (UID: \"d971e795-94ee-4fbc-809a-9d605c2e7b29\") " Oct 31 00:12:20.935206 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:20.934997 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-openshift-ca\") pod \"d971e795-94ee-4fbc-809a-9d605c2e7b29\" (UID: \"d971e795-94ee-4fbc-809a-9d605c2e7b29\") " Oct 31 00:12:20.935396 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:20.935373 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-openshift-ca" pod "d971e795-94ee-4fbc-809a-9d605c2e7b29" (UID: "d971e795-94ee-4fbc-809a-9d605c2e7b29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:20.937182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:20.937161 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-xks-ca" pod "d971e795-94ee-4fbc-809a-9d605c2e7b29" (UID: "d971e795-94ee-4fbc-809a-9d605c2e7b29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:20.937301 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:20.937284 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d971e795-94ee-4fbc-809a-9d605c2e7b29-kube-api-access-fzv5z" pod "d971e795-94ee-4fbc-809a-9d605c2e7b29" (UID: "d971e795-94ee-4fbc-809a-9d605c2e7b29"). InnerVolumeSpecName "kube-api-access-fzv5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:21.036764 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:21.036733 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:21.036764 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:21.036762 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzv5z\" (UniqueName: \"kubernetes.io/projected/d971e795-94ee-4fbc-809a-9d605c2e7b29-kube-api-access-fzv5z\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:21.036936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:21.036772 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d971e795-94ee-4fbc-809a-9d605c2e7b29-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:21.623547 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:21.623515 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" Oct 31 00:12:21.623721 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:21.623522 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk" event={"ID":"d971e795-94ee-4fbc-809a-9d605c2e7b29","Type":"ContainerDied","Data":"0eb9e7a828795415cfe658b0089b1ecd9d3d7ea9f3ea8c2907c8512434fafead"} Oct 31 00:12:21.623721 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:21.623630 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eb9e7a828795415cfe658b0089b1ecd9d3d7ea9f3ea8c2907c8512434fafead" Oct 31 00:12:29.221586 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.221538 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj"] Oct 31 00:12:29.222182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.221977 7570 cpu_manager.go:414] "containerMap: removing container" podUID="d971e795-94ee-4fbc-809a-9d605c2e7b29" containerName="revoke-keys" Oct 31 00:12:29.222182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.221991 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d971e795-94ee-4fbc-809a-9d605c2e7b29" containerName="revoke-keys" Oct 31 00:12:29.222182 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.222130 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="d971e795-94ee-4fbc-809a-9d605c2e7b29" containerName="revoke-keys" Oct 31 00:12:29.225814 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.225788 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.229742 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.229724 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c4c1ab-openshift-default-dockercfg-mqng8\"" Oct 31 00:12:29.238880 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.238858 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj"] Oct 31 00:12:29.320909 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.320865 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-credential-socket\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.321151 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.321111 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8db88698-da51-408b-af17-59c4f5af6aca-istio-podinfo\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.321263 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.321179 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-envoy\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.321263 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.321220 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-istio-token\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.321405 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.321262 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-certs\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.321405 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.321318 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8db88698-da51-408b-af17-59c4f5af6aca-istiod-ca-cert\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.321405 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.321349 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-socket\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.321588 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.321401 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4kfq\" (UniqueName: \"kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-kube-api-access-g4kfq\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.321588 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.321434 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-data\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.412083 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:12:29.412044 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:12:29.422325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.422296 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8db88698-da51-408b-af17-59c4f5af6aca-istiod-ca-cert\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.422535 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.422342 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-socket\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.422535 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.422422 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4kfq\" (UniqueName: \"kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-kube-api-access-g4kfq\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.422660 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.422531 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-data\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.422660 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.422578 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-credential-socket\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.422776 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.422717 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8db88698-da51-408b-af17-59c4f5af6aca-istio-podinfo\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.422776 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.422757 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-envoy\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.422885 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.422804 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-istio-token\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.422885 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.422852 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-certs\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.423492 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.423189 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8db88698-da51-408b-af17-59c4f5af6aca-istiod-ca-cert\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.423492 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.423422 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-socket\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.423689 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.423531 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-data\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.423689 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.423557 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-credential-socket\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.424039 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.424016 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-certs\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.426339 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.426310 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-envoy\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.426700 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.426590 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8db88698-da51-408b-af17-59c4f5af6aca-istio-podinfo\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.435259 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.435215 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4kfq\" (UniqueName: \"kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-kube-api-access-g4kfq\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.435259 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.435241 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-istio-token\") pod \"e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.541882 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.541749 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:29.825914 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.825891 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj"] Oct 31 00:12:29.827738 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:12:29.827712 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db88698_da51_408b_af17_59c4f5af6aca.slice/crio-7b8bbeaa73ded8cfd4ec66f37cb806a620793fb49b05ef1c000e740f1a2f1b54 WatchSource:0}: Error finding container 7b8bbeaa73ded8cfd4ec66f37cb806a620793fb49b05ef1c000e740f1a2f1b54: Status 404 returned error can't find the container with id 7b8bbeaa73ded8cfd4ec66f37cb806a620793fb49b05ef1c000e740f1a2f1b54 Oct 31 00:12:29.829906 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.829875 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:12:29.829980 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.829941 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:12:29.829980 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:29.829970 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:12:30.673725 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:30.673698 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" event={"ID":"8db88698-da51-408b-af17-59c4f5af6aca","Type":"ContainerStarted","Data":"b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d"} Oct 31 00:12:30.674136 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:30.673744 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" event={"ID":"8db88698-da51-408b-af17-59c4f5af6aca","Type":"ContainerStarted","Data":"7b8bbeaa73ded8cfd4ec66f37cb806a620793fb49b05ef1c000e740f1a2f1b54"} Oct 31 00:12:30.698846 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:30.698802 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" podStartSLOduration=1.698787066 podStartE2EDuration="1.698787066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:30.697357185 +0000 UTC m=+3162.216358274" watchObservedRunningTime="2026-10-31 00:12:30.698787066 +0000 UTC m=+3162.217788229" Oct 31 00:12:31.542382 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:31.542345 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:31.547434 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:31.547403 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:31.679710 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:31.679680 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:31.680704 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:31.680683 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:34.281190 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.281157 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5"] Oct 31 00:12:34.285509 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.285448 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" Oct 31 00:12:34.298679 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.298658 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c4c1ab\"" Oct 31 00:12:34.299822 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.299802 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5"] Oct 31 00:12:34.309672 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.309652 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b2075095-dockercfg-ck68d\"" Oct 31 00:12:34.332577 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.332552 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd"] Oct 31 00:12:34.336255 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.336236 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" Oct 31 00:12:34.347764 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.347727 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd"] Oct 31 00:12:34.369586 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.369556 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32199f2-6413-4625-ac30-d57b49b8276f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd\" (UID: \"d32199f2-6413-4625-ac30-d57b49b8276f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" Oct 31 00:12:34.369746 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.369690 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4rvk\" (UniqueName: \"kubernetes.io/projected/d32199f2-6413-4625-ac30-d57b49b8276f-kube-api-access-n4rvk\") pod \"payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd\" (UID: \"d32199f2-6413-4625-ac30-d57b49b8276f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" Oct 31 00:12:34.369792 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.369760 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20046265-076d-4f01-b425-4180bc369bdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5\" (UID: \"20046265-076d-4f01-b425-4180bc369bdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" Oct 31 00:12:34.369987 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.369963 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dptxw\" (UniqueName: \"kubernetes.io/projected/20046265-076d-4f01-b425-4180bc369bdb-kube-api-access-dptxw\") pod \"payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5\" (UID: \"20046265-076d-4f01-b425-4180bc369bdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" Oct 31 00:12:34.471190 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.471155 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4rvk\" (UniqueName: \"kubernetes.io/projected/d32199f2-6413-4625-ac30-d57b49b8276f-kube-api-access-n4rvk\") pod \"payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd\" (UID: \"d32199f2-6413-4625-ac30-d57b49b8276f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" Oct 31 00:12:34.471346 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.471215 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20046265-076d-4f01-b425-4180bc369bdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5\" (UID: \"20046265-076d-4f01-b425-4180bc369bdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" Oct 31 00:12:34.471406 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.471353 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dptxw\" (UniqueName: \"kubernetes.io/projected/20046265-076d-4f01-b425-4180bc369bdb-kube-api-access-dptxw\") pod \"payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5\" (UID: \"20046265-076d-4f01-b425-4180bc369bdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" Oct 31 00:12:34.471406 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.471397 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32199f2-6413-4625-ac30-d57b49b8276f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd\" (UID: \"d32199f2-6413-4625-ac30-d57b49b8276f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" Oct 31 00:12:34.471953 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.471929 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20046265-076d-4f01-b425-4180bc369bdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5\" (UID: \"20046265-076d-4f01-b425-4180bc369bdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" Oct 31 00:12:34.472081 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.472062 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32199f2-6413-4625-ac30-d57b49b8276f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd\" (UID: \"d32199f2-6413-4625-ac30-d57b49b8276f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" Oct 31 00:12:34.494066 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.494037 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4rvk\" (UniqueName: \"kubernetes.io/projected/d32199f2-6413-4625-ac30-d57b49b8276f-kube-api-access-n4rvk\") pod \"payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd\" (UID: \"d32199f2-6413-4625-ac30-d57b49b8276f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" Oct 31 00:12:34.494172 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.494038 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dptxw\" (UniqueName: \"kubernetes.io/projected/20046265-076d-4f01-b425-4180bc369bdb-kube-api-access-dptxw\") pod \"payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5\" (UID: \"20046265-076d-4f01-b425-4180bc369bdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" Oct 31 00:12:34.599434 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.599406 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" Oct 31 00:12:34.654045 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.653762 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" Oct 31 00:12:34.879645 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.879621 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5"] Oct 31 00:12:34.881570 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:12:34.881538 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20046265_076d_4f01_b425_4180bc369bdb.slice/crio-439ac4be24e65372f4d031da529d457b3441f8168de2fc8a4fdcf724b6af7726 WatchSource:0}: Error finding container 439ac4be24e65372f4d031da529d457b3441f8168de2fc8a4fdcf724b6af7726: Status 404 returned error can't find the container with id 439ac4be24e65372f4d031da529d457b3441f8168de2fc8a4fdcf724b6af7726 Oct 31 00:12:34.943888 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:34.943866 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd"] Oct 31 00:12:34.945172 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:12:34.945146 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32199f2_6413_4625_ac30_d57b49b8276f.slice/crio-ab848c563ba7dc1f9486ecb4b5e7fa613988ea806ad82d16bfa67e14893d7678 WatchSource:0}: Error finding container ab848c563ba7dc1f9486ecb4b5e7fa613988ea806ad82d16bfa67e14893d7678: Status 404 returned error can't find the container with id ab848c563ba7dc1f9486ecb4b5e7fa613988ea806ad82d16bfa67e14893d7678 Oct 31 00:12:35.704890 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:35.704851 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" event={"ID":"20046265-076d-4f01-b425-4180bc369bdb","Type":"ContainerStarted","Data":"2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3"} Oct 31 00:12:35.705385 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:35.704918 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" Oct 31 00:12:35.705385 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:35.704963 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" event={"ID":"20046265-076d-4f01-b425-4180bc369bdb","Type":"ContainerStarted","Data":"439ac4be24e65372f4d031da529d457b3441f8168de2fc8a4fdcf724b6af7726"} Oct 31 00:12:35.706661 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:35.706633 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" event={"ID":"d32199f2-6413-4625-ac30-d57b49b8276f","Type":"ContainerStarted","Data":"cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10"} Oct 31 00:12:35.706791 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:35.706684 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" event={"ID":"d32199f2-6413-4625-ac30-d57b49b8276f","Type":"ContainerStarted","Data":"ab848c563ba7dc1f9486ecb4b5e7fa613988ea806ad82d16bfa67e14893d7678"} Oct 31 00:12:35.706791 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:35.706715 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" Oct 31 00:12:35.725935 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:35.725876 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" podStartSLOduration=1.725857989 podStartE2EDuration="1.725857989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:35.72413908 +0000 UTC m=+3167.243140172" watchObservedRunningTime="2026-10-31 00:12:35.725857989 +0000 UTC m=+3167.244859080" Oct 31 00:12:35.745085 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:35.745049 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" podStartSLOduration=1.745037781 podStartE2EDuration="1.745037781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:35.743670109 +0000 UTC m=+3167.262671201" watchObservedRunningTime="2026-10-31 00:12:35.745037781 +0000 UTC m=+3167.264038870" Oct 31 00:12:43.610685 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:43.610662 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log" Oct 31 00:12:44.413336 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:44.413302 7570 scope.go:122] "RemoveContainer" containerID="64eb8eb10a549b03fe197616e24eae1977304e486a87c240a47a91cb341a8684" Oct 31 00:12:44.424630 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:44.424604 7570 scope.go:122] "RemoveContainer" containerID="40dcebdf055fcfd90b7f6e9a4944dc8f7f3ce4daef6002b149047d71736da467" Oct 31 00:12:44.435000 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:44.434979 7570 scope.go:122] "RemoveContainer" containerID="e2f966c9f488e25d24c13f6a147bd7b336a88d40f41259de8a0fff6649e3d363" Oct 31 00:12:44.445030 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:44.445007 7570 scope.go:122] "RemoveContainer" containerID="4015fab92b382ade63c02ce21e5ee6d9c1f855a08a44ca8ed5ce6261e51804c9" Oct 31 00:12:45.616414 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:45.616388 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log" Oct 31 00:12:46.681874 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:46.681842 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 31 00:12:46.682295 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:46.681904 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 31 00:12:46.715422 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:46.715399 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" Oct 31 00:12:46.715627 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:46.715613 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" Oct 31 00:12:46.772256 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:46.772221 7570 generic.go:356] "Generic (PLEG): container finished" podID="8db88698-da51-408b-af17-59c4f5af6aca" containerID="b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d" exitCode=0 Oct 31 00:12:46.772451 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:46.772293 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" event={"ID":"8db88698-da51-408b-af17-59c4f5af6aca","Type":"ContainerDied","Data":"b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d"} Oct 31 00:12:46.772792 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:46.772771 7570 scope.go:122] "RemoveContainer" containerID="b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d" Oct 31 00:12:46.774268 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:46.774237 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:12:46.774385 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:46.774317 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:12:46.774385 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:46.774370 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:12:47.780399 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:47.780355 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" event={"ID":"8db88698-da51-408b-af17-59c4f5af6aca","Type":"ContainerStarted","Data":"d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5"} Oct 31 00:12:48.541986 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:48.541947 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:48.542249 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:48.542195 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 31 00:12:48.542325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:48.542250 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 31 00:12:49.542639 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:49.542610 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 31 00:12:49.543012 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:49.542666 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 31 00:12:50.542406 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:50.542371 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 31 00:12:50.542606 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:50.542425 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 31 00:12:51.542608 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:51.542578 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 31 00:12:51.543030 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:51.542631 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 31 00:12:52.542551 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:52.542514 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 31 00:12:52.542733 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:52.542568 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 31 00:12:53.547005 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:53.546975 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:53.547436 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:53.547393 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:53.548204 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:12:53.548184 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:12:59.412034 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:12:59.412000 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:13:19.023259 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:19.023227 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk"] Oct 31 00:13:19.027346 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:19.027324 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mrsbk"] Oct 31 00:13:20.602276 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.602240 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4cd85c7-3856-436e-a326-d000dcc14e5a" path="/var/lib/kubelet/pods/c4cd85c7-3856-436e-a326-d000dcc14e5a/volumes" Oct 31 00:13:20.603776 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.603753 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj"] Oct 31 00:13:20.607521 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.607507 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:20.613912 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.613884 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:13:20.614052 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.613952 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:13:20.614052 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.613985 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:13:20.614159 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.614094 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:13:20.616803 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.616783 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj"] Oct 31 00:13:20.669798 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.669761 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6fafe5-jwlwj\" (UID: \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:20.669946 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.669806 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6fafe5-jwlwj\" (UID: \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:20.669946 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.669885 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbjjx\" (UniqueName: \"kubernetes.io/projected/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-kube-api-access-xbjjx\") pod \"maas-api-revoke-sub-9d6fafe5-jwlwj\" (UID: \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:20.770932 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.770899 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6fafe5-jwlwj\" (UID: \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:20.771093 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.770939 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6fafe5-jwlwj\" (UID: \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:20.771093 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.771070 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbjjx\" (UniqueName: \"kubernetes.io/projected/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-kube-api-access-xbjjx\") pod \"maas-api-revoke-sub-9d6fafe5-jwlwj\" (UID: \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:20.771842 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.771815 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6fafe5-jwlwj\" (UID: \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:20.773561 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.773538 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6fafe5-jwlwj\" (UID: \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:20.783189 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.783164 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbjjx\" (UniqueName: \"kubernetes.io/projected/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-kube-api-access-xbjjx\") pod \"maas-api-revoke-sub-9d6fafe5-jwlwj\" (UID: \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:20.922276 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:20.922194 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:21.202698 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:21.202673 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj"] Oct 31 00:13:21.205033 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:13:21.205008 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9c9461_8de7_44f7_bbff_4df18afe7bfc.slice/crio-2eaf666fdff17d215aa3bf3e6a8e2c2ea58d61b263a9cb03bffeb3130deb6ec1 WatchSource:0}: Error finding container 2eaf666fdff17d215aa3bf3e6a8e2c2ea58d61b263a9cb03bffeb3130deb6ec1: Status 404 returned error can't find the container with id 2eaf666fdff17d215aa3bf3e6a8e2c2ea58d61b263a9cb03bffeb3130deb6ec1 Oct 31 00:13:22.016804 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:22.016765 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" event={"ID":"0f9c9461-8de7-44f7-bbff-4df18afe7bfc","Type":"ContainerStarted","Data":"f65e56feff1d0eb313fdd4063cf4b72b9a321279efcca2b3295604649b2e4083"} Oct 31 00:13:22.017259 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:22.016818 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" event={"ID":"0f9c9461-8de7-44f7-bbff-4df18afe7bfc","Type":"ContainerStarted","Data":"2eaf666fdff17d215aa3bf3e6a8e2c2ea58d61b263a9cb03bffeb3130deb6ec1"} Oct 31 00:13:22.035643 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:22.035585 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" podStartSLOduration=2.03556503 podStartE2EDuration="2.03556503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:22.034188392 +0000 UTC m=+3213.553189483" watchObservedRunningTime="2026-10-31 00:13:22.03556503 +0000 UTC m=+3213.554566120" Oct 31 00:13:23.023163 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:23.023121 7570 generic.go:356] "Generic (PLEG): container finished" podID="0f9c9461-8de7-44f7-bbff-4df18afe7bfc" containerID="f65e56feff1d0eb313fdd4063cf4b72b9a321279efcca2b3295604649b2e4083" exitCode=0 Oct 31 00:13:23.023600 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:23.023205 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" event={"ID":"0f9c9461-8de7-44f7-bbff-4df18afe7bfc","Type":"ContainerDied","Data":"f65e56feff1d0eb313fdd4063cf4b72b9a321279efcca2b3295604649b2e4083"} Oct 31 00:13:24.172841 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:24.172818 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:24.308329 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:24.308244 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-kube-api-access-xbjjx\" (UniqueName: \"kubernetes.io/projected/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-kube-api-access-xbjjx\") pod \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\" (UID: \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\") " Oct 31 00:13:24.308529 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:24.308330 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-xks-ca\") pod \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\" (UID: \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\") " Oct 31 00:13:24.308529 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:24.308374 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-openshift-ca\") pod \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\" (UID: \"0f9c9461-8de7-44f7-bbff-4df18afe7bfc\") " Oct 31 00:13:24.308755 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:24.308732 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-openshift-ca" pod "0f9c9461-8de7-44f7-bbff-4df18afe7bfc" (UID: "0f9c9461-8de7-44f7-bbff-4df18afe7bfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:24.310572 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:24.310550 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-xks-ca" pod "0f9c9461-8de7-44f7-bbff-4df18afe7bfc" (UID: "0f9c9461-8de7-44f7-bbff-4df18afe7bfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:24.310642 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:24.310624 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-kube-api-access-xbjjx" pod "0f9c9461-8de7-44f7-bbff-4df18afe7bfc" (UID: "0f9c9461-8de7-44f7-bbff-4df18afe7bfc"). InnerVolumeSpecName "kube-api-access-xbjjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:24.409985 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:24.409950 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:24.409985 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:24.409979 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:24.409985 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:24.409989 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbjjx\" (UniqueName: \"kubernetes.io/projected/0f9c9461-8de7-44f7-bbff-4df18afe7bfc-kube-api-access-xbjjx\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:25.035901 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:25.035824 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" Oct 31 00:13:25.036048 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:25.035823 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj" event={"ID":"0f9c9461-8de7-44f7-bbff-4df18afe7bfc","Type":"ContainerDied","Data":"2eaf666fdff17d215aa3bf3e6a8e2c2ea58d61b263a9cb03bffeb3130deb6ec1"} Oct 31 00:13:25.036048 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:25.035943 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eaf666fdff17d215aa3bf3e6a8e2c2ea58d61b263a9cb03bffeb3130deb6ec1" Oct 31 00:13:29.024060 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:29.024028 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt"] Oct 31 00:13:29.026388 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:29.026364 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rxgdt"] Oct 31 00:13:29.412161 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:13:29.412117 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:13:30.601112 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:30.601076 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e70e07-c686-4ca0-94e0-b8b37a3d2058" path="/var/lib/kubelet/pods/c4e70e07-c686-4ca0-94e0-b8b37a3d2058/volumes" Oct 31 00:13:31.602491 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.602440 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg"] Oct 31 00:13:31.602929 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.602915 7570 cpu_manager.go:414] "containerMap: removing container" podUID="0f9c9461-8de7-44f7-bbff-4df18afe7bfc" containerName="revoke-keys" Oct 31 00:13:31.602977 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.602931 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9c9461-8de7-44f7-bbff-4df18afe7bfc" containerName="revoke-keys" Oct 31 00:13:31.603048 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.603038 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9c9461-8de7-44f7-bbff-4df18afe7bfc" containerName="revoke-keys" Oct 31 00:13:31.606712 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.606696 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:31.614537 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.614508 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:13:31.614690 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.614658 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:13:31.615017 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.614871 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:13:31.615075 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.615045 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:13:31.630511 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.622089 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg"] Oct 31 00:13:31.687207 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.687176 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg\" (UID: \"0e868d81-2bba-48b1-8e81-571e3930c452\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:31.687367 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.687347 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzrwh\" (UniqueName: \"kubernetes.io/projected/0e868d81-2bba-48b1-8e81-571e3930c452-kube-api-access-gzrwh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg\" (UID: \"0e868d81-2bba-48b1-8e81-571e3930c452\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:31.687413 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.687393 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg\" (UID: \"0e868d81-2bba-48b1-8e81-571e3930c452\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:31.788346 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.788311 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzrwh\" (UniqueName: \"kubernetes.io/projected/0e868d81-2bba-48b1-8e81-571e3930c452-kube-api-access-gzrwh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg\" (UID: \"0e868d81-2bba-48b1-8e81-571e3930c452\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:31.788581 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.788355 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg\" (UID: \"0e868d81-2bba-48b1-8e81-571e3930c452\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:31.788581 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.788396 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg\" (UID: \"0e868d81-2bba-48b1-8e81-571e3930c452\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:31.789116 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.789087 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg\" (UID: \"0e868d81-2bba-48b1-8e81-571e3930c452\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:31.790928 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.790904 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg\" (UID: \"0e868d81-2bba-48b1-8e81-571e3930c452\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:31.803506 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.803484 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzrwh\" (UniqueName: \"kubernetes.io/projected/0e868d81-2bba-48b1-8e81-571e3930c452-kube-api-access-gzrwh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg\" (UID: \"0e868d81-2bba-48b1-8e81-571e3930c452\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:31.930677 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:31.930589 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:32.221049 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:32.221015 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg"] Oct 31 00:13:32.223181 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:13:32.223142 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e868d81_2bba_48b1_8e81_571e3930c452.slice/crio-021b7d2ee0143420947ad51040385094fa6e7d56da8892cddedd798353435ae2 WatchSource:0}: Error finding container 021b7d2ee0143420947ad51040385094fa6e7d56da8892cddedd798353435ae2: Status 404 returned error can't find the container with id 021b7d2ee0143420947ad51040385094fa6e7d56da8892cddedd798353435ae2 Oct 31 00:13:33.081586 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:33.081556 7570 generic.go:356] "Generic (PLEG): container finished" podID="0e868d81-2bba-48b1-8e81-571e3930c452" containerID="246b77730e443a3e495de7ec4e3d51ec7dacabeb2d7bd5c3f2053006bc74e205" exitCode=0 Oct 31 00:13:33.081936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:33.081644 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" event={"ID":"0e868d81-2bba-48b1-8e81-571e3930c452","Type":"ContainerDied","Data":"246b77730e443a3e495de7ec4e3d51ec7dacabeb2d7bd5c3f2053006bc74e205"} Oct 31 00:13:33.081936 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:33.081698 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" event={"ID":"0e868d81-2bba-48b1-8e81-571e3930c452","Type":"ContainerStarted","Data":"021b7d2ee0143420947ad51040385094fa6e7d56da8892cddedd798353435ae2"} Oct 31 00:13:34.225098 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:34.225067 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:34.316109 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:34.316080 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e868d81-2bba-48b1-8e81-571e3930c452-kube-api-access-gzrwh\" (UniqueName: \"kubernetes.io/projected/0e868d81-2bba-48b1-8e81-571e3930c452-kube-api-access-gzrwh\") pod \"0e868d81-2bba-48b1-8e81-571e3930c452\" (UID: \"0e868d81-2bba-48b1-8e81-571e3930c452\") " Oct 31 00:13:34.316291 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:34.316194 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-xks-ca\") pod \"0e868d81-2bba-48b1-8e81-571e3930c452\" (UID: \"0e868d81-2bba-48b1-8e81-571e3930c452\") " Oct 31 00:13:34.316291 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:34.316235 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-openshift-ca\") pod \"0e868d81-2bba-48b1-8e81-571e3930c452\" (UID: \"0e868d81-2bba-48b1-8e81-571e3930c452\") " Oct 31 00:13:34.316674 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:34.316651 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-openshift-ca" pod "0e868d81-2bba-48b1-8e81-571e3930c452" (UID: "0e868d81-2bba-48b1-8e81-571e3930c452"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:34.318408 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:34.318379 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e868d81-2bba-48b1-8e81-571e3930c452-kube-api-access-gzrwh" pod "0e868d81-2bba-48b1-8e81-571e3930c452" (UID: "0e868d81-2bba-48b1-8e81-571e3930c452"). InnerVolumeSpecName "kube-api-access-gzrwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:34.318521 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:34.318443 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-xks-ca" pod "0e868d81-2bba-48b1-8e81-571e3930c452" (UID: "0e868d81-2bba-48b1-8e81-571e3930c452"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:34.417949 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:34.417868 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzrwh\" (UniqueName: \"kubernetes.io/projected/0e868d81-2bba-48b1-8e81-571e3930c452-kube-api-access-gzrwh\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:34.417949 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:34.417898 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:34.417949 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:34.417909 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e868d81-2bba-48b1-8e81-571e3930c452-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:35.094859 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:35.094827 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" Oct 31 00:13:35.094859 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:35.094845 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg" event={"ID":"0e868d81-2bba-48b1-8e81-571e3930c452","Type":"ContainerDied","Data":"021b7d2ee0143420947ad51040385094fa6e7d56da8892cddedd798353435ae2"} Oct 31 00:13:35.095071 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:35.094887 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="021b7d2ee0143420947ad51040385094fa6e7d56da8892cddedd798353435ae2" Oct 31 00:13:41.707220 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:41.707184 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd"] Oct 31 00:13:41.721300 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:41.721268 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5"] Oct 31 00:13:42.296873 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:42.296841 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg"] Oct 31 00:13:42.299377 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:42.299350 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4c1ab-xbpbg"] Oct 31 00:13:42.601420 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:42.601386 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e868d81-2bba-48b1-8e81-571e3930c452" path="/var/lib/kubelet/pods/0e868d81-2bba-48b1-8e81-571e3930c452/volumes" Oct 31 00:13:44.486688 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:44.486655 7570 scope.go:122] "RemoveContainer" containerID="ea82d2844320ed986eb1fc6c01320fe5fc742837f956232e9702b145ad34ab3a" Oct 31 00:13:44.496088 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:44.496068 7570 scope.go:122] "RemoveContainer" containerID="836eab80ee4a2571f478fdf97b85d2a7a2e9a21bd2e4b72d58537c68dbb332dd" Oct 31 00:13:44.509665 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:44.509641 7570 scope.go:122] "RemoveContainer" containerID="fc525553f7b1be6ea05ee46ca784942ddd063b432968cb540dde06be97080cc1" Oct 31 00:13:46.708238 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:46.708171 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" podUID="d32199f2-6413-4625-ac30-d57b49b8276f" containerName="payload-processing" containerID="cri-o://cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10" gracePeriod=25 Oct 31 00:13:46.714667 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:46.714641 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" podUID="d32199f2-6413-4625-ac30-d57b49b8276f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:13:46.722304 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:46.722275 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" podUID="20046265-076d-4f01-b425-4180bc369bdb" containerName="payload-pre-processing" containerID="cri-o://2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3" gracePeriod=25 Oct 31 00:13:49.280906 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:49.280874 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj"] Oct 31 00:13:49.281313 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:49.281115 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" containerID="cri-o://d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5" gracePeriod=30 Oct 31 00:13:53.550347 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:53.550310 7570 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:13:53.550751 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:53.550364 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:13:54.542495 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.542454 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:13:54.619121 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619096 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-socket\") pod \"8db88698-da51-408b-af17-59c4f5af6aca\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " Oct 31 00:13:54.619602 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619141 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-istio-token\" (UniqueName: \"kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-istio-token\") pod \"8db88698-da51-408b-af17-59c4f5af6aca\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " Oct 31 00:13:54.619602 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619171 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-data\") pod \"8db88698-da51-408b-af17-59c4f5af6aca\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " Oct 31 00:13:54.619602 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619236 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-credential-socket\") pod \"8db88698-da51-408b-af17-59c4f5af6aca\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " Oct 31 00:13:54.619602 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619288 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-kube-api-access-g4kfq\" (UniqueName: \"kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-kube-api-access-g4kfq\") pod \"8db88698-da51-408b-af17-59c4f5af6aca\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " Oct 31 00:13:54.619602 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619351 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-certs\") pod \"8db88698-da51-408b-af17-59c4f5af6aca\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " Oct 31 00:13:54.619602 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619384 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-socket" pod "8db88698-da51-408b-af17-59c4f5af6aca" (UID: "8db88698-da51-408b-af17-59c4f5af6aca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:13:54.619602 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619417 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-envoy\") pod \"8db88698-da51-408b-af17-59c4f5af6aca\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " Oct 31 00:13:54.619602 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619495 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8db88698-da51-408b-af17-59c4f5af6aca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8db88698-da51-408b-af17-59c4f5af6aca-istio-podinfo\") pod \"8db88698-da51-408b-af17-59c4f5af6aca\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " Oct 31 00:13:54.619602 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619523 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8db88698-da51-408b-af17-59c4f5af6aca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8db88698-da51-408b-af17-59c4f5af6aca-istiod-ca-cert\") pod \"8db88698-da51-408b-af17-59c4f5af6aca\" (UID: \"8db88698-da51-408b-af17-59c4f5af6aca\") " Oct 31 00:13:54.619602 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619570 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-data" pod "8db88698-da51-408b-af17-59c4f5af6aca" (UID: "8db88698-da51-408b-af17-59c4f5af6aca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:13:54.619602 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619585 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-credential-socket" pod "8db88698-da51-408b-af17-59c4f5af6aca" (UID: "8db88698-da51-408b-af17-59c4f5af6aca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:13:54.620082 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619900 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-certs" pod "8db88698-da51-408b-af17-59c4f5af6aca" (UID: "8db88698-da51-408b-af17-59c4f5af6aca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:13:54.620082 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619949 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:54.620082 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619968 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:54.620082 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.619982 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:54.620254 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.620011 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db88698-da51-408b-af17-59c4f5af6aca-istiod-ca-cert" pod "8db88698-da51-408b-af17-59c4f5af6aca" (UID: "8db88698-da51-408b-af17-59c4f5af6aca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:54.621810 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.621781 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-kube-api-access-g4kfq" pod "8db88698-da51-408b-af17-59c4f5af6aca" (UID: "8db88698-da51-408b-af17-59c4f5af6aca"). InnerVolumeSpecName "kube-api-access-g4kfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:54.621937 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.621896 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-envoy" pod "8db88698-da51-408b-af17-59c4f5af6aca" (UID: "8db88698-da51-408b-af17-59c4f5af6aca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:13:54.622110 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.622094 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-istio-token" pod "8db88698-da51-408b-af17-59c4f5af6aca" (UID: "8db88698-da51-408b-af17-59c4f5af6aca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:54.622268 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.622254 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8db88698-da51-408b-af17-59c4f5af6aca-istio-podinfo" pod "8db88698-da51-408b-af17-59c4f5af6aca" (UID: "8db88698-da51-408b-af17-59c4f5af6aca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:13:54.721513 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.721451 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:54.721513 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.721507 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4kfq\" (UniqueName: \"kubernetes.io/projected/8db88698-da51-408b-af17-59c4f5af6aca-kube-api-access-g4kfq\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:54.721513 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.721518 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:54.721739 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.721526 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8db88698-da51-408b-af17-59c4f5af6aca-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:54.721739 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.721534 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8db88698-da51-408b-af17-59c4f5af6aca-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:54.721739 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:54.721542 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8db88698-da51-408b-af17-59c4f5af6aca-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:55.211222 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:55.211187 7570 generic.go:356] "Generic (PLEG): container finished" podID="8db88698-da51-408b-af17-59c4f5af6aca" containerID="d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5" exitCode=0 Oct 31 00:13:55.211538 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:55.211251 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" Oct 31 00:13:55.211538 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:55.211278 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" event={"ID":"8db88698-da51-408b-af17-59c4f5af6aca","Type":"ContainerDied","Data":"d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5"} Oct 31 00:13:55.211538 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:55.211348 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj" event={"ID":"8db88698-da51-408b-af17-59c4f5af6aca","Type":"ContainerDied","Data":"7b8bbeaa73ded8cfd4ec66f37cb806a620793fb49b05ef1c000e740f1a2f1b54"} Oct 31 00:13:55.211538 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:55.211374 7570 scope.go:122] "RemoveContainer" containerID="d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5" Oct 31 00:13:55.222286 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:55.222270 7570 scope.go:122] "RemoveContainer" containerID="b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d" Oct 31 00:13:55.231344 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:55.231328 7570 scope.go:122] "RemoveContainer" containerID="d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5" Oct 31 00:13:55.231825 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:13:55.231785 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5\": container with ID starting with d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5 not found: ID does not exist" containerID="d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5" Oct 31 00:13:55.231921 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:55.231856 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5"} err="failed to get container status \"d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5\": rpc error: code = NotFound desc = could not find container \"d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5\": container with ID starting with d71cf28f0a1cab66bba289c6930069aa9a602cc52a1ff765a6b1b16d19858bf5 not found: ID does not exist" Oct 31 00:13:55.231921 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:55.231891 7570 scope.go:122] "RemoveContainer" containerID="b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d" Oct 31 00:13:55.232840 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:13:55.232821 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d\": container with ID starting with b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d not found: ID does not exist" containerID="b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d" Oct 31 00:13:55.232923 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:55.232847 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d"} err="failed to get container status \"b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d\": rpc error: code = NotFound desc = could not find container \"b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d\": container with ID starting with b623c71db7abd702f53017fcb37136271576d7f9a3af7d4a5bca3ca6a60ce32d not found: ID does not exist" Oct 31 00:13:55.234857 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:55.234828 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj"] Oct 31 00:13:55.240879 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:55.240857 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4c1ab-openshift-default-7c6fdb6bb5m8klj"] Oct 31 00:13:56.601686 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:56.601647 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8db88698-da51-408b-af17-59c4f5af6aca" path="/var/lib/kubelet/pods/8db88698-da51-408b-af17-59c4f5af6aca/volumes" Oct 31 00:13:56.714102 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:56.714061 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" podUID="20046265-076d-4f01-b425-4180bc369bdb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:13:56.714317 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:13:56.714290 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" podUID="d32199f2-6413-4625-ac30-d57b49b8276f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:13:59.411983 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:13:59.411944 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:14:06.713909 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:14:06.713873 7570 logging.go:55] [core] [Channel #3799 SubChannel #3800]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.233:50052", ServerName: "10.134.0.233:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.233:50052: connect: connection refused" Oct 31 00:14:06.714651 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:06.714450 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" podUID="20046265-076d-4f01-b425-4180bc369bdb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:14:06.972612 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:06.972549 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" Oct 31 00:14:07.000758 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.000733 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" Oct 31 00:14:07.051039 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.051008 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20046265-076d-4f01-b425-4180bc369bdb-kube-api-access-dptxw\" (UniqueName: \"kubernetes.io/projected/20046265-076d-4f01-b425-4180bc369bdb-kube-api-access-dptxw\") pod \"20046265-076d-4f01-b425-4180bc369bdb\" (UID: \"20046265-076d-4f01-b425-4180bc369bdb\") " Oct 31 00:14:07.051191 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.051072 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32199f2-6413-4625-ac30-d57b49b8276f-kube-api-access-n4rvk\" (UniqueName: \"kubernetes.io/projected/d32199f2-6413-4625-ac30-d57b49b8276f-kube-api-access-n4rvk\") pod \"d32199f2-6413-4625-ac30-d57b49b8276f\" (UID: \"d32199f2-6413-4625-ac30-d57b49b8276f\") " Oct 31 00:14:07.051191 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.051129 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20046265-076d-4f01-b425-4180bc369bdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20046265-076d-4f01-b425-4180bc369bdb-plugins-config-volume\") pod \"20046265-076d-4f01-b425-4180bc369bdb\" (UID: \"20046265-076d-4f01-b425-4180bc369bdb\") " Oct 31 00:14:07.051311 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.051214 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32199f2-6413-4625-ac30-d57b49b8276f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32199f2-6413-4625-ac30-d57b49b8276f-plugins-config-volume\") pod \"d32199f2-6413-4625-ac30-d57b49b8276f\" (UID: \"d32199f2-6413-4625-ac30-d57b49b8276f\") " Oct 31 00:14:07.051562 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.051540 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20046265-076d-4f01-b425-4180bc369bdb-plugins-config-volume" pod "20046265-076d-4f01-b425-4180bc369bdb" (UID: "20046265-076d-4f01-b425-4180bc369bdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:07.051656 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.051637 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32199f2-6413-4625-ac30-d57b49b8276f-plugins-config-volume" pod "d32199f2-6413-4625-ac30-d57b49b8276f" (UID: "d32199f2-6413-4625-ac30-d57b49b8276f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:07.053229 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.053203 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20046265-076d-4f01-b425-4180bc369bdb-kube-api-access-dptxw" pod "20046265-076d-4f01-b425-4180bc369bdb" (UID: "20046265-076d-4f01-b425-4180bc369bdb"). InnerVolumeSpecName "kube-api-access-dptxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:07.053410 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.053311 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32199f2-6413-4625-ac30-d57b49b8276f-kube-api-access-n4rvk" pod "d32199f2-6413-4625-ac30-d57b49b8276f" (UID: "d32199f2-6413-4625-ac30-d57b49b8276f"). InnerVolumeSpecName "kube-api-access-n4rvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:07.152532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.152502 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dptxw\" (UniqueName: \"kubernetes.io/projected/20046265-076d-4f01-b425-4180bc369bdb-kube-api-access-dptxw\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:07.152532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.152529 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4rvk\" (UniqueName: \"kubernetes.io/projected/d32199f2-6413-4625-ac30-d57b49b8276f-kube-api-access-n4rvk\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:07.152698 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.152542 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20046265-076d-4f01-b425-4180bc369bdb-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:07.152698 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.152555 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32199f2-6413-4625-ac30-d57b49b8276f-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:07.311176 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.311085 7570 generic.go:356] "Generic (PLEG): container finished" podID="20046265-076d-4f01-b425-4180bc369bdb" containerID="2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3" exitCode=0 Oct 31 00:14:07.311176 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.311142 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" Oct 31 00:14:07.311176 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.311165 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" event={"ID":"20046265-076d-4f01-b425-4180bc369bdb","Type":"ContainerDied","Data":"2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3"} Oct 31 00:14:07.311455 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.311214 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5" event={"ID":"20046265-076d-4f01-b425-4180bc369bdb","Type":"ContainerDied","Data":"439ac4be24e65372f4d031da529d457b3441f8168de2fc8a4fdcf724b6af7726"} Oct 31 00:14:07.311455 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.311235 7570 scope.go:122] "RemoveContainer" containerID="2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3" Oct 31 00:14:07.312724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.312694 7570 generic.go:356] "Generic (PLEG): container finished" podID="d32199f2-6413-4625-ac30-d57b49b8276f" containerID="cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10" exitCode=0 Oct 31 00:14:07.312841 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.312729 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" event={"ID":"d32199f2-6413-4625-ac30-d57b49b8276f","Type":"ContainerDied","Data":"cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10"} Oct 31 00:14:07.312841 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.312746 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" Oct 31 00:14:07.312841 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.312789 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" event={"ID":"d32199f2-6413-4625-ac30-d57b49b8276f","Type":"ContainerDied","Data":"ab848c563ba7dc1f9486ecb4b5e7fa613988ea806ad82d16bfa67e14893d7678"} Oct 31 00:14:07.321728 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.321709 7570 scope.go:122] "RemoveContainer" containerID="2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3" Oct 31 00:14:07.321979 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:14:07.321964 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3\": container with ID starting with 2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3 not found: ID does not exist" containerID="2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3" Oct 31 00:14:07.322048 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.321986 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3"} err="failed to get container status \"2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3\": rpc error: code = NotFound desc = could not find container \"2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3\": container with ID starting with 2fc11c6e5b02ee7a6adab0af4431868869933dbd2b4dcab929caa1f4f1f04de3 not found: ID does not exist" Oct 31 00:14:07.322048 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.322000 7570 scope.go:122] "RemoveContainer" containerID="cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10" Oct 31 00:14:07.330566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.330551 7570 scope.go:122] "RemoveContainer" containerID="cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10" Oct 31 00:14:07.330837 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:14:07.330818 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10\": container with ID starting with cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10 not found: ID does not exist" containerID="cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10" Oct 31 00:14:07.330884 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.330844 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10"} err="failed to get container status \"cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10\": rpc error: code = NotFound desc = could not find container \"cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10\": container with ID starting with cd8f8e14bd3836eb86cc071bf0b020c3797ee10c42d3384f0df6cba616c5fc10 not found: ID does not exist" Oct 31 00:14:07.338075 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.338047 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd"] Oct 31 00:14:07.342078 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.342055 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd"] Oct 31 00:14:07.353707 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.353689 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5"] Oct 31 00:14:07.358798 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.358774 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4c1ab-68675ndjs5"] Oct 31 00:14:07.713659 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:07.713621 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4c1ab-6b797b8fd2h8pd" podUID="d32199f2-6413-4625-ac30-d57b49b8276f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.233:50052\" within 1s: context deadline exceeded" Oct 31 00:14:08.601677 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:08.601639 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20046265-076d-4f01-b425-4180bc369bdb" path="/var/lib/kubelet/pods/20046265-076d-4f01-b425-4180bc369bdb/volumes" Oct 31 00:14:08.602225 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:08.602202 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32199f2-6413-4625-ac30-d57b49b8276f" path="/var/lib/kubelet/pods/d32199f2-6413-4625-ac30-d57b49b8276f/volumes" Oct 31 00:14:10.152980 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.152944 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq"] Oct 31 00:14:10.153653 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153631 7570 cpu_manager.go:414] "containerMap: removing container" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" Oct 31 00:14:10.153705 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153658 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" Oct 31 00:14:10.153705 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153686 7570 cpu_manager.go:414] "containerMap: removing container" podUID="20046265-076d-4f01-b425-4180bc369bdb" containerName="payload-pre-processing" Oct 31 00:14:10.153705 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153694 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20046265-076d-4f01-b425-4180bc369bdb" containerName="payload-pre-processing" Oct 31 00:14:10.153801 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153710 7570 cpu_manager.go:414] "containerMap: removing container" podUID="d32199f2-6413-4625-ac30-d57b49b8276f" containerName="payload-processing" Oct 31 00:14:10.153801 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153718 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32199f2-6413-4625-ac30-d57b49b8276f" containerName="payload-processing" Oct 31 00:14:10.153801 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153732 7570 cpu_manager.go:414] "containerMap: removing container" podUID="0e868d81-2bba-48b1-8e81-571e3930c452" containerName="revoke-keys" Oct 31 00:14:10.153801 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153739 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e868d81-2bba-48b1-8e81-571e3930c452" containerName="revoke-keys" Oct 31 00:14:10.153913 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153873 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="20046265-076d-4f01-b425-4180bc369bdb" containerName="payload-pre-processing" Oct 31 00:14:10.153913 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153886 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32199f2-6413-4625-ac30-d57b49b8276f" containerName="payload-processing" Oct 31 00:14:10.153913 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153894 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e868d81-2bba-48b1-8e81-571e3930c452" containerName="revoke-keys" Oct 31 00:14:10.153913 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153905 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" Oct 31 00:14:10.154025 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.153924 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" Oct 31 00:14:10.159080 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.159057 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.164526 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.164502 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq"] Oct 31 00:14:10.164678 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.164653 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a30f2db8-kserve-self-signed-certs\"" Oct 31 00:14:10.281986 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.281952 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-model-cache\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.282147 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.282005 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-home\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.282147 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.282127 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-tmp-dir\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.282226 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.282151 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-dshm\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.282226 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.282177 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7586df77-199a-43c4-8851-2f4ce1f1d355-tls-certs\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.282302 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.282286 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdbd\" (UniqueName: \"kubernetes.io/projected/7586df77-199a-43c4-8851-2f4ce1f1d355-kube-api-access-5bdbd\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.383255 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.383217 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-tmp-dir\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.383255 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.383259 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-dshm\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.383558 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.383298 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7586df77-199a-43c4-8851-2f4ce1f1d355-tls-certs\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.383558 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.383358 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdbd\" (UniqueName: \"kubernetes.io/projected/7586df77-199a-43c4-8851-2f4ce1f1d355-kube-api-access-5bdbd\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.383558 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.383401 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-model-cache\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.383558 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.383450 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-home\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.383750 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.383699 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-tmp-dir\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.383816 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.383763 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-model-cache\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.383906 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.383887 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-home\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.385857 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.385838 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-dshm\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.386063 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.386042 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7586df77-199a-43c4-8851-2f4ce1f1d355-tls-certs\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.396011 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.395988 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdbd\" (UniqueName: \"kubernetes.io/projected/7586df77-199a-43c4-8851-2f4ce1f1d355-kube-api-access-5bdbd\") pod \"e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.478063 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.477998 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:10.755478 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:10.755438 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq"] Oct 31 00:14:10.757526 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:14:10.757497 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7586df77_199a_43c4_8851_2f4ce1f1d355.slice/crio-55ba9f1e0f8dfd794e33214dcdb46d37676c353fa9c9ba261f1a4999cd4d6ff4 WatchSource:0}: Error finding container 55ba9f1e0f8dfd794e33214dcdb46d37676c353fa9c9ba261f1a4999cd4d6ff4: Status 404 returned error can't find the container with id 55ba9f1e0f8dfd794e33214dcdb46d37676c353fa9c9ba261f1a4999cd4d6ff4 Oct 31 00:14:11.344223 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:11.344186 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" event={"ID":"7586df77-199a-43c4-8851-2f4ce1f1d355","Type":"ContainerStarted","Data":"088ec67f2bcf3b3dd0f89bb945ef41fe86d0109c1d870937c413855b5f239421"} Oct 31 00:14:11.344223 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:11.344228 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" event={"ID":"7586df77-199a-43c4-8851-2f4ce1f1d355","Type":"ContainerStarted","Data":"55ba9f1e0f8dfd794e33214dcdb46d37676c353fa9c9ba261f1a4999cd4d6ff4"} Oct 31 00:14:11.364965 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:11.364919 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" podStartSLOduration=1.3649062060000001 podStartE2EDuration="1.364906206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:11.363313405 +0000 UTC m=+3262.882314492" watchObservedRunningTime="2026-10-31 00:14:11.364906206 +0000 UTC m=+3262.883907296" Oct 31 00:14:11.479169 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:11.479136 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:20.479146 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:20.479065 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:20.494291 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:20.494264 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:21.418773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:21.418738 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:27.345643 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.345605 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx"] Oct 31 00:14:27.346127 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.346111 7570 cpu_manager.go:414] "containerMap: removing container" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" Oct 31 00:14:27.346171 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.346131 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8db88698-da51-408b-af17-59c4f5af6aca" containerName="istio-proxy" Oct 31 00:14:27.349504 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.349486 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:27.355374 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.355348 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:14:27.357174 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.357153 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:14:27.357394 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.357360 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:14:27.358097 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.358053 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:14:27.361502 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.361479 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx"] Oct 31 00:14:27.507278 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.507245 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4dbhx\" (UID: \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:27.507493 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.507316 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqr6c\" (UniqueName: \"kubernetes.io/projected/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-kube-api-access-zqr6c\") pod \"maas-api-revoke-sub-452b6d7b-4dbhx\" (UID: \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:27.507493 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.507434 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4dbhx\" (UID: \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:27.608854 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.608773 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4dbhx\" (UID: \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:27.608854 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.608835 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4dbhx\" (UID: \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:27.609041 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.608874 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqr6c\" (UniqueName: \"kubernetes.io/projected/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-kube-api-access-zqr6c\") pod \"maas-api-revoke-sub-452b6d7b-4dbhx\" (UID: \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:27.609446 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.609427 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4dbhx\" (UID: \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:27.611523 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.611501 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4dbhx\" (UID: \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:27.621609 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.621579 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqr6c\" (UniqueName: \"kubernetes.io/projected/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-kube-api-access-zqr6c\") pod \"maas-api-revoke-sub-452b6d7b-4dbhx\" (UID: \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:27.668503 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.668478 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:27.945207 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:27.945185 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx"] Oct 31 00:14:27.947173 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:14:27.947148 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c40240_9d81_48f6_9b7c_f1f256d6a76b.slice/crio-d3f4e0a5f4572c4cdd579995b9dd9ee86518c1df68a6fee2222f2b6837536a7f WatchSource:0}: Error finding container d3f4e0a5f4572c4cdd579995b9dd9ee86518c1df68a6fee2222f2b6837536a7f: Status 404 returned error can't find the container with id d3f4e0a5f4572c4cdd579995b9dd9ee86518c1df68a6fee2222f2b6837536a7f Oct 31 00:14:28.454055 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:28.453938 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" event={"ID":"a1c40240-9d81-48f6-9b7c-f1f256d6a76b","Type":"ContainerStarted","Data":"2209d59a92739e25d1861b1e3ffe854b48956d2ac62c1255bb919d4f84a0680c"} Oct 31 00:14:28.454537 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:28.454058 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" event={"ID":"a1c40240-9d81-48f6-9b7c-f1f256d6a76b","Type":"ContainerStarted","Data":"d3f4e0a5f4572c4cdd579995b9dd9ee86518c1df68a6fee2222f2b6837536a7f"} Oct 31 00:14:28.472634 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:28.472569 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" podStartSLOduration=1.472552735 podStartE2EDuration="1.472552735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:28.471675357 +0000 UTC m=+3279.990676447" watchObservedRunningTime="2026-10-31 00:14:28.472552735 +0000 UTC m=+3279.991553826" Oct 31 00:14:29.418102 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:14:29.418060 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:14:29.469489 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:29.468724 7570 generic.go:356] "Generic (PLEG): container finished" podID="a1c40240-9d81-48f6-9b7c-f1f256d6a76b" containerID="2209d59a92739e25d1861b1e3ffe854b48956d2ac62c1255bb919d4f84a0680c" exitCode=0 Oct 31 00:14:29.469489 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:29.468916 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" event={"ID":"a1c40240-9d81-48f6-9b7c-f1f256d6a76b","Type":"ContainerDied","Data":"2209d59a92739e25d1861b1e3ffe854b48956d2ac62c1255bb919d4f84a0680c"} Oct 31 00:14:30.894486 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:30.886066 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:30.976503 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:30.975323 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-kube-api-access-zqr6c\" (UniqueName: \"kubernetes.io/projected/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-kube-api-access-zqr6c\") pod \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\" (UID: \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\") " Oct 31 00:14:30.976503 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:30.975498 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-xks-ca\") pod \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\" (UID: \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\") " Oct 31 00:14:30.976503 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:30.975656 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-openshift-ca\") pod \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\" (UID: \"a1c40240-9d81-48f6-9b7c-f1f256d6a76b\") " Oct 31 00:14:30.976503 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:30.976289 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-openshift-ca" pod "a1c40240-9d81-48f6-9b7c-f1f256d6a76b" (UID: "a1c40240-9d81-48f6-9b7c-f1f256d6a76b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:30.977948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:30.977921 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-kube-api-access-zqr6c" pod "a1c40240-9d81-48f6-9b7c-f1f256d6a76b" (UID: "a1c40240-9d81-48f6-9b7c-f1f256d6a76b"). InnerVolumeSpecName "kube-api-access-zqr6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:30.978869 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:30.978844 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-xks-ca" pod "a1c40240-9d81-48f6-9b7c-f1f256d6a76b" (UID: "a1c40240-9d81-48f6-9b7c-f1f256d6a76b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:31.077283 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:31.077251 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:31.077283 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:31.077281 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:31.077502 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:31.077293 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqr6c\" (UniqueName: \"kubernetes.io/projected/a1c40240-9d81-48f6-9b7c-f1f256d6a76b-kube-api-access-zqr6c\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:31.482383 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:31.482351 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" Oct 31 00:14:31.482572 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:31.482398 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx" event={"ID":"a1c40240-9d81-48f6-9b7c-f1f256d6a76b","Type":"ContainerDied","Data":"d3f4e0a5f4572c4cdd579995b9dd9ee86518c1df68a6fee2222f2b6837536a7f"} Oct 31 00:14:31.482572 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:31.482443 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3f4e0a5f4572c4cdd579995b9dd9ee86518c1df68a6fee2222f2b6837536a7f" Oct 31 00:14:39.468081 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:39.468049 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq"] Oct 31 00:14:41.085379 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.085330 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7"] Oct 31 00:14:41.086083 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.086060 7570 cpu_manager.go:414] "containerMap: removing container" podUID="a1c40240-9d81-48f6-9b7c-f1f256d6a76b" containerName="revoke-keys" Oct 31 00:14:41.086166 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.086087 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c40240-9d81-48f6-9b7c-f1f256d6a76b" containerName="revoke-keys" Oct 31 00:14:41.086252 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.086237 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c40240-9d81-48f6-9b7c-f1f256d6a76b" containerName="revoke-keys" Oct 31 00:14:41.090499 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.090474 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:41.097592 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.097552 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7"] Oct 31 00:14:41.101648 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.100374 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:14:41.101648 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.100524 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:14:41.101648 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.100559 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:14:41.101648 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.100597 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:14:41.182592 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.182553 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xdzx7\" (UID: \"4bc96048-571a-42f1-89d1-e5c754d1ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:41.182795 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.182695 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xdzx7\" (UID: \"4bc96048-571a-42f1-89d1-e5c754d1ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:41.182864 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.182809 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjgxn\" (UniqueName: \"kubernetes.io/projected/4bc96048-571a-42f1-89d1-e5c754d1ee63-kube-api-access-cjgxn\") pod \"maas-api-revoke-sub-3e31a506-xdzx7\" (UID: \"4bc96048-571a-42f1-89d1-e5c754d1ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:41.284506 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.284446 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xdzx7\" (UID: \"4bc96048-571a-42f1-89d1-e5c754d1ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:41.284699 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.284598 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjgxn\" (UniqueName: \"kubernetes.io/projected/4bc96048-571a-42f1-89d1-e5c754d1ee63-kube-api-access-cjgxn\") pod \"maas-api-revoke-sub-3e31a506-xdzx7\" (UID: \"4bc96048-571a-42f1-89d1-e5c754d1ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:41.284766 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.284703 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xdzx7\" (UID: \"4bc96048-571a-42f1-89d1-e5c754d1ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:41.285555 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.285529 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xdzx7\" (UID: \"4bc96048-571a-42f1-89d1-e5c754d1ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:41.287216 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.287191 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xdzx7\" (UID: \"4bc96048-571a-42f1-89d1-e5c754d1ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:41.299095 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.299065 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjgxn\" (UniqueName: \"kubernetes.io/projected/4bc96048-571a-42f1-89d1-e5c754d1ee63-kube-api-access-cjgxn\") pod \"maas-api-revoke-sub-3e31a506-xdzx7\" (UID: \"4bc96048-571a-42f1-89d1-e5c754d1ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:41.423717 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.423619 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:41.845540 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:41.844331 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7"] Oct 31 00:14:42.564556 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:42.564057 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" event={"ID":"4bc96048-571a-42f1-89d1-e5c754d1ee63","Type":"ContainerStarted","Data":"ab38fb48c9a4a82bfb51779dea82429850efe28eb06b77762204de916ceda0d5"} Oct 31 00:14:42.564556 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:42.564125 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" event={"ID":"4bc96048-571a-42f1-89d1-e5c754d1ee63","Type":"ContainerStarted","Data":"5eb39b11ed9ef337d5f5ed48e3f9c7be3bdc20294ffef42653f86058a70f42ce"} Oct 31 00:14:42.601290 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:42.601231 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" podStartSLOduration=1.601211607 podStartE2EDuration="1.601211607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:42.599528137 +0000 UTC m=+3294.118529228" watchObservedRunningTime="2026-10-31 00:14:42.601211607 +0000 UTC m=+3294.120212697" Oct 31 00:14:43.574259 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:43.574153 7570 generic.go:356] "Generic (PLEG): container finished" podID="4bc96048-571a-42f1-89d1-e5c754d1ee63" containerID="ab38fb48c9a4a82bfb51779dea82429850efe28eb06b77762204de916ceda0d5" exitCode=0 Oct 31 00:14:43.574259 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:43.574237 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" event={"ID":"4bc96048-571a-42f1-89d1-e5c754d1ee63","Type":"ContainerDied","Data":"ab38fb48c9a4a82bfb51779dea82429850efe28eb06b77762204de916ceda0d5"} Oct 31 00:14:44.560509 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:44.553213 7570 scope.go:122] "RemoveContainer" containerID="1ca607f5cdcf10104bb0d64c3af48e2992ab26c9f64dd614e2774abfd6de041f" Oct 31 00:14:44.568638 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:44.568615 7570 scope.go:122] "RemoveContainer" containerID="fb3a5cf59f673390cb2f08e97080ecdef3237dd3eee4df37a259ae53d1a1e2e2" Oct 31 00:14:44.577855 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:44.577835 7570 scope.go:122] "RemoveContainer" containerID="181c9a34794ea7d9bb1c80eaef83da93dbb26b6a8f0205dc658456cc6491d888" Oct 31 00:14:44.623940 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:44.622609 7570 scope.go:122] "RemoveContainer" containerID="20d7b1fdd89a654b2a4dc5a8e7b33ccc32dae2a2584403afb5d83020f23ea0cf" Oct 31 00:14:45.055943 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.055594 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:45.130005 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.129972 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-openshift-ca\") pod \"4bc96048-571a-42f1-89d1-e5c754d1ee63\" (UID: \"4bc96048-571a-42f1-89d1-e5c754d1ee63\") " Oct 31 00:14:45.130302 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.130270 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-xks-ca\") pod \"4bc96048-571a-42f1-89d1-e5c754d1ee63\" (UID: \"4bc96048-571a-42f1-89d1-e5c754d1ee63\") " Oct 31 00:14:45.130617 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.130596 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc96048-571a-42f1-89d1-e5c754d1ee63-kube-api-access-cjgxn\" (UniqueName: \"kubernetes.io/projected/4bc96048-571a-42f1-89d1-e5c754d1ee63-kube-api-access-cjgxn\") pod \"4bc96048-571a-42f1-89d1-e5c754d1ee63\" (UID: \"4bc96048-571a-42f1-89d1-e5c754d1ee63\") " Oct 31 00:14:45.142066 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.137883 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-openshift-ca" pod "4bc96048-571a-42f1-89d1-e5c754d1ee63" (UID: "4bc96048-571a-42f1-89d1-e5c754d1ee63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:45.166608 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.166576 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-xks-ca" pod "4bc96048-571a-42f1-89d1-e5c754d1ee63" (UID: "4bc96048-571a-42f1-89d1-e5c754d1ee63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:45.170222 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.170192 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc96048-571a-42f1-89d1-e5c754d1ee63-kube-api-access-cjgxn" pod "4bc96048-571a-42f1-89d1-e5c754d1ee63" (UID: "4bc96048-571a-42f1-89d1-e5c754d1ee63"). InnerVolumeSpecName "kube-api-access-cjgxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:45.243541 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.243241 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-openshift-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:45.243541 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.243277 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc96048-571a-42f1-89d1-e5c754d1ee63-maas-api-xks-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:45.243541 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.243293 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjgxn\" (UniqueName: \"kubernetes.io/projected/4bc96048-571a-42f1-89d1-e5c754d1ee63-kube-api-access-cjgxn\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:45.611977 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.611940 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" event={"ID":"4bc96048-571a-42f1-89d1-e5c754d1ee63","Type":"ContainerDied","Data":"5eb39b11ed9ef337d5f5ed48e3f9c7be3bdc20294ffef42653f86058a70f42ce"} Oct 31 00:14:45.612594 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.612572 7570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eb39b11ed9ef337d5f5ed48e3f9c7be3bdc20294ffef42653f86058a70f42ce" Oct 31 00:14:45.612728 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:45.612669 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7" Oct 31 00:14:52.023294 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:52.023259 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j"] Oct 31 00:14:52.028965 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:52.028934 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1376e47c-dnv8j"] Oct 31 00:14:52.631486 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:52.631398 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4540c6a7-be99-4a24-afdb-1a85613bed36" path="/var/lib/kubelet/pods/4540c6a7-be99-4a24-afdb-1a85613bed36/volumes" Oct 31 00:14:54.490589 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.490530 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" podUID="7586df77-199a-43c4-8851-2f4ce1f1d355" containerName="main" containerID="cri-o://088ec67f2bcf3b3dd0f89bb945ef41fe86d0109c1d870937c413855b5f239421" gracePeriod=45 Oct 31 00:14:54.679388 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.679354 7570 generic.go:356] "Generic (PLEG): container finished" podID="7586df77-199a-43c4-8851-2f4ce1f1d355" containerID="088ec67f2bcf3b3dd0f89bb945ef41fe86d0109c1d870937c413855b5f239421" exitCode=0 Oct 31 00:14:54.679579 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.679402 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" event={"ID":"7586df77-199a-43c4-8851-2f4ce1f1d355","Type":"ContainerDied","Data":"088ec67f2bcf3b3dd0f89bb945ef41fe86d0109c1d870937c413855b5f239421"} Oct 31 00:14:54.747441 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.747373 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:54.872994 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.872951 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-dshm\") pod \"7586df77-199a-43c4-8851-2f4ce1f1d355\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " Oct 31 00:14:54.873160 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.873074 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-model-cache\") pod \"7586df77-199a-43c4-8851-2f4ce1f1d355\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " Oct 31 00:14:54.873160 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.873116 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-home\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-home\") pod \"7586df77-199a-43c4-8851-2f4ce1f1d355\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " Oct 31 00:14:54.873276 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.873161 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7586df77-199a-43c4-8851-2f4ce1f1d355-tls-certs\" (UniqueName: \"kubernetes.io/secret/7586df77-199a-43c4-8851-2f4ce1f1d355-tls-certs\") pod \"7586df77-199a-43c4-8851-2f4ce1f1d355\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " Oct 31 00:14:54.873276 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.873243 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7586df77-199a-43c4-8851-2f4ce1f1d355-kube-api-access-5bdbd\" (UniqueName: \"kubernetes.io/projected/7586df77-199a-43c4-8851-2f4ce1f1d355-kube-api-access-5bdbd\") pod \"7586df77-199a-43c4-8851-2f4ce1f1d355\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " Oct 31 00:14:54.873427 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.873399 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-model-cache" pod "7586df77-199a-43c4-8851-2f4ce1f1d355" (UID: "7586df77-199a-43c4-8851-2f4ce1f1d355"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:14:54.873803 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.873753 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-home" pod "7586df77-199a-43c4-8851-2f4ce1f1d355" (UID: "7586df77-199a-43c4-8851-2f4ce1f1d355"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:14:54.874071 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.873801 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-tmp-dir\") pod \"7586df77-199a-43c4-8851-2f4ce1f1d355\" (UID: \"7586df77-199a-43c4-8851-2f4ce1f1d355\") " Oct 31 00:14:54.874439 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.874390 7570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-model-cache\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:54.874439 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.874417 7570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-home\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:54.874439 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.874415 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-tmp-dir" pod "7586df77-199a-43c4-8851-2f4ce1f1d355" (UID: "7586df77-199a-43c4-8851-2f4ce1f1d355"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:14:54.875721 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.875695 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7586df77-199a-43c4-8851-2f4ce1f1d355-tls-certs" pod "7586df77-199a-43c4-8851-2f4ce1f1d355" (UID: "7586df77-199a-43c4-8851-2f4ce1f1d355"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:54.875905 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.875877 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-dshm" pod "7586df77-199a-43c4-8851-2f4ce1f1d355" (UID: "7586df77-199a-43c4-8851-2f4ce1f1d355"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:14:54.876019 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.875971 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7586df77-199a-43c4-8851-2f4ce1f1d355-kube-api-access-5bdbd" pod "7586df77-199a-43c4-8851-2f4ce1f1d355" (UID: "7586df77-199a-43c4-8851-2f4ce1f1d355"). InnerVolumeSpecName "kube-api-access-5bdbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:54.976201 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.976164 7570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7586df77-199a-43c4-8851-2f4ce1f1d355-tls-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:54.976201 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.976205 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bdbd\" (UniqueName: \"kubernetes.io/projected/7586df77-199a-43c4-8851-2f4ce1f1d355-kube-api-access-5bdbd\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:54.976402 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.976224 7570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-tmp-dir\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:54.976402 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:54.976237 7570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7586df77-199a-43c4-8851-2f4ce1f1d355-dshm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:55.686113 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:55.686076 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" event={"ID":"7586df77-199a-43c4-8851-2f4ce1f1d355","Type":"ContainerDied","Data":"55ba9f1e0f8dfd794e33214dcdb46d37676c353fa9c9ba261f1a4999cd4d6ff4"} Oct 31 00:14:55.686743 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:55.686721 7570 scope.go:122] "RemoveContainer" containerID="088ec67f2bcf3b3dd0f89bb945ef41fe86d0109c1d870937c413855b5f239421" Oct 31 00:14:55.686980 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:55.686247 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq" Oct 31 00:14:55.771112 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:55.771087 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq"] Oct 31 00:14:55.778346 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:55.778324 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a30f2db8-kserve-b78bbd4cc-bggsq"] Oct 31 00:14:56.626576 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:14:56.622792 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7586df77-199a-43c4-8851-2f4ce1f1d355" path="/var/lib/kubelet/pods/7586df77-199a-43c4-8851-2f4ce1f1d355/volumes" Oct 31 00:14:59.411272 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:14:59.411239 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:15:00.138403 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.138368 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz"] Oct 31 00:15:00.138869 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.138855 7570 cpu_manager.go:414] "containerMap: removing container" podUID="4bc96048-571a-42f1-89d1-e5c754d1ee63" containerName="revoke-keys" Oct 31 00:15:00.138955 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.138871 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc96048-571a-42f1-89d1-e5c754d1ee63" containerName="revoke-keys" Oct 31 00:15:00.138955 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.138904 7570 cpu_manager.go:414] "containerMap: removing container" podUID="7586df77-199a-43c4-8851-2f4ce1f1d355" containerName="main" Oct 31 00:15:00.138955 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.138910 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7586df77-199a-43c4-8851-2f4ce1f1d355" containerName="main" Oct 31 00:15:00.139046 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.138995 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7586df77-199a-43c4-8851-2f4ce1f1d355" containerName="main" Oct 31 00:15:00.139046 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.139012 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc96048-571a-42f1-89d1-e5c754d1ee63" containerName="revoke-keys" Oct 31 00:15:00.143390 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.143373 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" Oct 31 00:15:00.148828 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.148804 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\"" Oct 31 00:15:00.149155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.149122 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:15:00.149294 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.149173 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:15:00.152561 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.152537 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz"] Oct 31 00:15:00.254502 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.254445 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-gwgkz\" (UID: \"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" Oct 31 00:15:00.254660 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.254635 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrhx\" (UniqueName: \"kubernetes.io/projected/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-kube-api-access-dbrhx\") pod \"maas-api-key-cleanup-29890095-gwgkz\" (UID: \"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" Oct 31 00:15:00.355276 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.355246 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrhx\" (UniqueName: \"kubernetes.io/projected/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-kube-api-access-dbrhx\") pod \"maas-api-key-cleanup-29890095-gwgkz\" (UID: \"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" Oct 31 00:15:00.355414 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.355304 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-gwgkz\" (UID: \"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" Oct 31 00:15:00.355913 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.355895 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-gwgkz\" (UID: \"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" Oct 31 00:15:00.370072 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.370042 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrhx\" (UniqueName: \"kubernetes.io/projected/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-kube-api-access-dbrhx\") pod \"maas-api-key-cleanup-29890095-gwgkz\" (UID: \"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" Oct 31 00:15:00.458084 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.458014 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" Oct 31 00:15:00.734591 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:00.734567 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz"] Oct 31 00:15:00.739116 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:15:00.739069 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ae6c6b_cb32_42f8_87f4_98d3cbd5e819.slice/crio-6f24a957770edc9183dd57376e4a9d318a3ea09a940d120201e0ffc6584de165 WatchSource:0}: Error finding container 6f24a957770edc9183dd57376e4a9d318a3ea09a940d120201e0ffc6584de165: Status 404 returned error can't find the container with id 6f24a957770edc9183dd57376e4a9d318a3ea09a940d120201e0ffc6584de165 Oct 31 00:15:01.732311 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:01.732283 7570 generic.go:356] "Generic (PLEG): container finished" podID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" containerID="a2300c371c39baeca501c1e51ab48c5e045f0563a8f59186538ad9f7b7decc08" exitCode=60 Oct 31 00:15:01.732688 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:01.732368 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" event={"ID":"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819","Type":"ContainerDied","Data":"a2300c371c39baeca501c1e51ab48c5e045f0563a8f59186538ad9f7b7decc08"} Oct 31 00:15:01.732688 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:01.732421 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" event={"ID":"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819","Type":"ContainerStarted","Data":"6f24a957770edc9183dd57376e4a9d318a3ea09a940d120201e0ffc6584de165"} Oct 31 00:15:01.732688 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:01.732584 7570 scope.go:122] "RemoveContainer" containerID="a2300c371c39baeca501c1e51ab48c5e045f0563a8f59186538ad9f7b7decc08" Oct 31 00:15:02.739438 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:02.739408 7570 generic.go:356] "Generic (PLEG): container finished" podID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" containerID="f989770731d5f5d9b6b4c66a99cbfd9f2897629ce5e3f521592f87a538b801f6" exitCode=60 Oct 31 00:15:02.739844 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:02.739497 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" event={"ID":"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819","Type":"ContainerDied","Data":"f989770731d5f5d9b6b4c66a99cbfd9f2897629ce5e3f521592f87a538b801f6"} Oct 31 00:15:02.739844 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:02.739567 7570 scope.go:122] "RemoveContainer" containerID="a2300c371c39baeca501c1e51ab48c5e045f0563a8f59186538ad9f7b7decc08" Oct 31 00:15:02.739844 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:02.739806 7570 scope.go:122] "RemoveContainer" containerID="f989770731d5f5d9b6b4c66a99cbfd9f2897629ce5e3f521592f87a538b801f6" Oct 31 00:15:02.740070 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:15:02.740050 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-gwgkz_odh-ai-gateway-infra(53ae6c6b-cb32-42f8-87f4-98d3cbd5e819)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" Oct 31 00:15:03.747051 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:03.747014 7570 scope.go:122] "RemoveContainer" containerID="f989770731d5f5d9b6b4c66a99cbfd9f2897629ce5e3f521592f87a538b801f6" Oct 31 00:15:03.747506 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:15:03.747248 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-gwgkz_odh-ai-gateway-infra(53ae6c6b-cb32-42f8-87f4-98d3cbd5e819)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" Oct 31 00:15:13.833748 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.833713 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9"] Oct 31 00:15:13.838687 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.838668 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:13.847932 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.847906 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 00:15:13.847932 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.847913 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:15:13.848101 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.848084 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-59scr\"" Oct 31 00:15:13.852172 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.852147 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9"] Oct 31 00:15:13.892076 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.892038 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qzln\" (UniqueName: \"kubernetes.io/projected/621bf2bc-d847-406e-b916-ed91c38a6fa1-kube-api-access-8qzln\") pod \"maas-api-55477bdbbc-wdcg9\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:13.892215 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.892153 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-maas-api-tls\") pod \"maas-api-55477bdbbc-wdcg9\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:13.892215 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.892204 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-metrics-certs\") pod \"maas-api-55477bdbbc-wdcg9\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:13.892295 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.892249 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621bf2bc-d847-406e-b916-ed91c38a6fa1-openshift-service-ca\") pod \"maas-api-55477bdbbc-wdcg9\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:13.992895 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.992866 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621bf2bc-d847-406e-b916-ed91c38a6fa1-openshift-service-ca\") pod \"maas-api-55477bdbbc-wdcg9\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:13.993038 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.992947 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qzln\" (UniqueName: \"kubernetes.io/projected/621bf2bc-d847-406e-b916-ed91c38a6fa1-kube-api-access-8qzln\") pod \"maas-api-55477bdbbc-wdcg9\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:13.993038 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.992979 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-maas-api-tls\") pod \"maas-api-55477bdbbc-wdcg9\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:13.993038 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.993000 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-metrics-certs\") pod \"maas-api-55477bdbbc-wdcg9\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:13.993720 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.993623 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621bf2bc-d847-406e-b916-ed91c38a6fa1-openshift-service-ca\") pod \"maas-api-55477bdbbc-wdcg9\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:13.995620 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.995597 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-maas-api-tls\") pod \"maas-api-55477bdbbc-wdcg9\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:13.995737 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:13.995718 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-metrics-certs\") pod \"maas-api-55477bdbbc-wdcg9\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:14.005655 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:14.005625 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qzln\" (UniqueName: \"kubernetes.io/projected/621bf2bc-d847-406e-b916-ed91c38a6fa1-kube-api-access-8qzln\") pod \"maas-api-55477bdbbc-wdcg9\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:14.155807 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:14.155725 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:14.439229 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:14.439203 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9"] Oct 31 00:15:14.440805 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:15:14.440782 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621bf2bc_d847_406e_b916_ed91c38a6fa1.slice/crio-c2f4abff89e8c7d335b6d4624124aa1b36409cc1f92b3cd084861509be56625c WatchSource:0}: Error finding container c2f4abff89e8c7d335b6d4624124aa1b36409cc1f92b3cd084861509be56625c: Status 404 returned error can't find the container with id c2f4abff89e8c7d335b6d4624124aa1b36409cc1f92b3cd084861509be56625c Oct 31 00:15:14.442148 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:14.442127 7570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:15:14.595280 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:14.595254 7570 scope.go:122] "RemoveContainer" containerID="f989770731d5f5d9b6b4c66a99cbfd9f2897629ce5e3f521592f87a538b801f6" Oct 31 00:15:14.811283 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:14.811256 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" event={"ID":"621bf2bc-d847-406e-b916-ed91c38a6fa1","Type":"ContainerStarted","Data":"c2f4abff89e8c7d335b6d4624124aa1b36409cc1f92b3cd084861509be56625c"} Oct 31 00:15:15.818557 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:15.818454 7570 generic.go:356] "Generic (PLEG): container finished" podID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" containerID="5bb3680fca8a716bc885bdcd31c75fabac933b35c936bf2c401910a24a3a9d21" exitCode=60 Oct 31 00:15:15.818557 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:15.818496 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" event={"ID":"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819","Type":"ContainerDied","Data":"5bb3680fca8a716bc885bdcd31c75fabac933b35c936bf2c401910a24a3a9d21"} Oct 31 00:15:15.819081 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:15.818564 7570 scope.go:122] "RemoveContainer" containerID="f989770731d5f5d9b6b4c66a99cbfd9f2897629ce5e3f521592f87a538b801f6" Oct 31 00:15:15.819081 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:15.818903 7570 scope.go:122] "RemoveContainer" containerID="5bb3680fca8a716bc885bdcd31c75fabac933b35c936bf2c401910a24a3a9d21" Oct 31 00:15:15.819179 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:15:15.819163 7570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-gwgkz_odh-ai-gateway-infra(53ae6c6b-cb32-42f8-87f4-98d3cbd5e819)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" Oct 31 00:15:15.820262 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:15.820240 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" event={"ID":"621bf2bc-d847-406e-b916-ed91c38a6fa1","Type":"ContainerStarted","Data":"cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22"} Oct 31 00:15:15.820378 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:15.820332 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:15.899208 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:15.899159 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" podStartSLOduration=2.281999142 podStartE2EDuration="2.89914546s" totalImagesPullingTime="617.146318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:15.896751585 +0000 UTC m=+3327.415752675" watchObservedRunningTime="2026-10-31 00:15:15.89914546 +0000 UTC m=+3327.418146549" Oct 31 00:15:16.880145 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:16.880107 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz"] Oct 31 00:15:17.017267 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:17.017244 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" Oct 31 00:15:17.120919 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:17.120889 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-kube-api-access-dbrhx\" (UniqueName: \"kubernetes.io/projected/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-kube-api-access-dbrhx\") pod \"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819\" (UID: \"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819\") " Oct 31 00:15:17.121084 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:17.120959 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-maas-api-service-ca\") pod \"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819\" (UID: \"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819\") " Oct 31 00:15:17.121333 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:17.121311 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-maas-api-service-ca" pod "53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" (UID: "53ae6c6b-cb32-42f8-87f4-98d3cbd5e819"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:17.123371 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:17.123344 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-kube-api-access-dbrhx" pod "53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" (UID: "53ae6c6b-cb32-42f8-87f4-98d3cbd5e819"). InnerVolumeSpecName "kube-api-access-dbrhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:17.222561 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:17.222490 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrhx\" (UniqueName: \"kubernetes.io/projected/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-kube-api-access-dbrhx\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:17.222561 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:17.222517 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819-maas-api-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:17.834024 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:17.833996 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" Oct 31 00:15:17.834024 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:17.834006 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz" event={"ID":"53ae6c6b-cb32-42f8-87f4-98d3cbd5e819","Type":"ContainerDied","Data":"6f24a957770edc9183dd57376e4a9d318a3ea09a940d120201e0ffc6584de165"} Oct 31 00:15:17.834267 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:17.834062 7570 scope.go:122] "RemoveContainer" containerID="5bb3680fca8a716bc885bdcd31c75fabac933b35c936bf2c401910a24a3a9d21" Oct 31 00:15:17.857958 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:17.857934 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz"] Oct 31 00:15:17.861809 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:17.861788 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-gwgkz"] Oct 31 00:15:18.605495 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:18.605440 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" path="/var/lib/kubelet/pods/53ae6c6b-cb32-42f8-87f4-98d3cbd5e819/volumes" Oct 31 00:15:21.400110 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.400076 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9"] Oct 31 00:15:21.400566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.400283 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" podUID="621bf2bc-d847-406e-b916-ed91c38a6fa1" containerName="maas-api" containerID="cri-o://cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22" gracePeriod=30 Oct 31 00:15:21.405390 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.405370 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:21.454708 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.454653 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48"] Oct 31 00:15:21.455370 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.455350 7570 cpu_manager.go:414] "containerMap: removing container" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" containerName="cleanup" Oct 31 00:15:21.455428 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.455374 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" containerName="cleanup" Oct 31 00:15:21.455428 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.455412 7570 cpu_manager.go:414] "containerMap: removing container" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" containerName="cleanup" Oct 31 00:15:21.455428 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.455421 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" containerName="cleanup" Oct 31 00:15:21.455535 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.455439 7570 cpu_manager.go:414] "containerMap: removing container" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" containerName="cleanup" Oct 31 00:15:21.455535 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.455449 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" containerName="cleanup" Oct 31 00:15:21.455592 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.455575 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" containerName="cleanup" Oct 31 00:15:21.455627 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.455592 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" containerName="cleanup" Oct 31 00:15:21.458992 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.458974 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.491134 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.491106 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48"] Oct 31 00:15:21.517734 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.517705 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt"] Oct 31 00:15:21.518734 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.518707 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ae6c6b-cb32-42f8-87f4-98d3cbd5e819" containerName="cleanup" Oct 31 00:15:21.522103 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.522083 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.525931 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.525902 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-48ebba5a-openshift-default-dockercfg-mdgl7\"" Oct 31 00:15:21.540416 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.540390 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt"] Oct 31 00:15:21.563986 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.563957 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1970122b-758e-45f0-8d17-1f132f893b05-openshift-service-ca\") pod \"maas-api-7f5c9bcb67-2tr48\" (UID: \"1970122b-758e-45f0-8d17-1f132f893b05\") " pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.563986 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.563990 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-credential-socket\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.564234 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.564014 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1970122b-758e-45f0-8d17-1f132f893b05-metrics-certs\") pod \"maas-api-7f5c9bcb67-2tr48\" (UID: \"1970122b-758e-45f0-8d17-1f132f893b05\") " pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.564234 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.564156 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-podinfo\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.564234 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.564197 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-data\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.564234 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.564233 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-socket\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.564436 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.564327 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbed3d8-69e3-4ab4-b309-8478c511571c-istiod-ca-cert\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.564436 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.564379 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-certs\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.564436 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.564411 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-token\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.564613 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.564444 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvlg\" (UniqueName: \"kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-kube-api-access-vjvlg\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.564613 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.564588 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1970122b-758e-45f0-8d17-1f132f893b05-maas-api-tls\") pod \"maas-api-7f5c9bcb67-2tr48\" (UID: \"1970122b-758e-45f0-8d17-1f132f893b05\") " pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.564722 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.564646 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8522f\" (UniqueName: \"kubernetes.io/projected/1970122b-758e-45f0-8d17-1f132f893b05-kube-api-access-8522f\") pod \"maas-api-7f5c9bcb67-2tr48\" (UID: \"1970122b-758e-45f0-8d17-1f132f893b05\") " pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.564775 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.564723 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-envoy\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.665682 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.665656 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-podinfo\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.665815 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.665716 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-data\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.665815 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.665768 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-socket\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.665815 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.665802 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbed3d8-69e3-4ab4-b309-8478c511571c-istiod-ca-cert\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.665983 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.665840 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-certs\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.665983 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.665868 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-token\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.665983 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.665902 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvlg\" (UniqueName: \"kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-kube-api-access-vjvlg\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.665983 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.665937 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1970122b-758e-45f0-8d17-1f132f893b05-maas-api-tls\") pod \"maas-api-7f5c9bcb67-2tr48\" (UID: \"1970122b-758e-45f0-8d17-1f132f893b05\") " pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.665983 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.665980 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8522f\" (UniqueName: \"kubernetes.io/projected/1970122b-758e-45f0-8d17-1f132f893b05-kube-api-access-8522f\") pod \"maas-api-7f5c9bcb67-2tr48\" (UID: \"1970122b-758e-45f0-8d17-1f132f893b05\") " pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.666221 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.666031 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-envoy\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.666221 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.666060 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1970122b-758e-45f0-8d17-1f132f893b05-openshift-service-ca\") pod \"maas-api-7f5c9bcb67-2tr48\" (UID: \"1970122b-758e-45f0-8d17-1f132f893b05\") " pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.666221 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.666085 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-credential-socket\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.666221 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.666103 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-data\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.666221 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.666119 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1970122b-758e-45f0-8d17-1f132f893b05-metrics-certs\") pod \"maas-api-7f5c9bcb67-2tr48\" (UID: \"1970122b-758e-45f0-8d17-1f132f893b05\") " pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.666490 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.666223 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-certs\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.666724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.666697 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-socket\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.666724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.666708 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbed3d8-69e3-4ab4-b309-8478c511571c-istiod-ca-cert\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.666901 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.666829 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-credential-socket\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.667138 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.667090 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1970122b-758e-45f0-8d17-1f132f893b05-openshift-service-ca\") pod \"maas-api-7f5c9bcb67-2tr48\" (UID: \"1970122b-758e-45f0-8d17-1f132f893b05\") " pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.668856 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.668836 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-podinfo\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.669204 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.669165 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-envoy\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.669400 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.669378 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1970122b-758e-45f0-8d17-1f132f893b05-metrics-certs\") pod \"maas-api-7f5c9bcb67-2tr48\" (UID: \"1970122b-758e-45f0-8d17-1f132f893b05\") " pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.669611 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.669596 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1970122b-758e-45f0-8d17-1f132f893b05-maas-api-tls\") pod \"maas-api-7f5c9bcb67-2tr48\" (UID: \"1970122b-758e-45f0-8d17-1f132f893b05\") " pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.672921 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.672905 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:21.678748 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.678726 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8522f\" (UniqueName: \"kubernetes.io/projected/1970122b-758e-45f0-8d17-1f132f893b05-kube-api-access-8522f\") pod \"maas-api-7f5c9bcb67-2tr48\" (UID: \"1970122b-758e-45f0-8d17-1f132f893b05\") " pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.678836 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.678768 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-token\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.679113 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.679095 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvlg\" (UniqueName: \"kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-kube-api-access-vjvlg\") pod \"e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.767378 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.767330 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621bf2bc-d847-406e-b916-ed91c38a6fa1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621bf2bc-d847-406e-b916-ed91c38a6fa1-openshift-service-ca\") pod \"621bf2bc-d847-406e-b916-ed91c38a6fa1\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " Oct 31 00:15:21.767599 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.767431 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-maas-api-tls\") pod \"621bf2bc-d847-406e-b916-ed91c38a6fa1\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " Oct 31 00:15:21.767599 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.767541 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621bf2bc-d847-406e-b916-ed91c38a6fa1-kube-api-access-8qzln\" (UniqueName: \"kubernetes.io/projected/621bf2bc-d847-406e-b916-ed91c38a6fa1-kube-api-access-8qzln\") pod \"621bf2bc-d847-406e-b916-ed91c38a6fa1\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " Oct 31 00:15:21.767599 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.767593 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-metrics-certs\") pod \"621bf2bc-d847-406e-b916-ed91c38a6fa1\" (UID: \"621bf2bc-d847-406e-b916-ed91c38a6fa1\") " Oct 31 00:15:21.767766 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.767744 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621bf2bc-d847-406e-b916-ed91c38a6fa1-openshift-service-ca" pod "621bf2bc-d847-406e-b916-ed91c38a6fa1" (UID: "621bf2bc-d847-406e-b916-ed91c38a6fa1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:21.768027 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.768009 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621bf2bc-d847-406e-b916-ed91c38a6fa1-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:21.769731 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.769707 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621bf2bc-d847-406e-b916-ed91c38a6fa1-kube-api-access-8qzln" pod "621bf2bc-d847-406e-b916-ed91c38a6fa1" (UID: "621bf2bc-d847-406e-b916-ed91c38a6fa1"). InnerVolumeSpecName "kube-api-access-8qzln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:21.769825 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.769757 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-metrics-certs" pod "621bf2bc-d847-406e-b916-ed91c38a6fa1" (UID: "621bf2bc-d847-406e-b916-ed91c38a6fa1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:15:21.769825 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.769769 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-maas-api-tls" pod "621bf2bc-d847-406e-b916-ed91c38a6fa1" (UID: "621bf2bc-d847-406e-b916-ed91c38a6fa1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:15:21.791761 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.791742 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:21.837069 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.837028 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:21.868811 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.868784 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:21.868918 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.868817 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qzln\" (UniqueName: \"kubernetes.io/projected/621bf2bc-d847-406e-b916-ed91c38a6fa1-kube-api-access-8qzln\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:21.868918 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.868837 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621bf2bc-d847-406e-b916-ed91c38a6fa1-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:21.872061 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.872032 7570 generic.go:356] "Generic (PLEG): container finished" podID="621bf2bc-d847-406e-b916-ed91c38a6fa1" containerID="cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22" exitCode=0 Oct 31 00:15:21.872181 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.872111 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" Oct 31 00:15:21.872181 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.872124 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" event={"ID":"621bf2bc-d847-406e-b916-ed91c38a6fa1","Type":"ContainerDied","Data":"cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22"} Oct 31 00:15:21.872299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.872219 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9" event={"ID":"621bf2bc-d847-406e-b916-ed91c38a6fa1","Type":"ContainerDied","Data":"c2f4abff89e8c7d335b6d4624124aa1b36409cc1f92b3cd084861509be56625c"} Oct 31 00:15:21.872299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.872255 7570 scope.go:122] "RemoveContainer" containerID="cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22" Oct 31 00:15:21.885691 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.885611 7570 scope.go:122] "RemoveContainer" containerID="cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22" Oct 31 00:15:21.885922 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:15:21.885899 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22\": container with ID starting with cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22 not found: ID does not exist" containerID="cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22" Oct 31 00:15:21.886053 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.885930 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22"} err="failed to get container status \"cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22\": rpc error: code = NotFound desc = could not find container \"cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22\": container with ID starting with cb3a09425ea5ad9dbb3690753dbd2899f17c0e03dd32fe81ec2c55757a3ceb22 not found: ID does not exist" Oct 31 00:15:21.900849 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.900780 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9"] Oct 31 00:15:21.904941 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:21.904880 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55477bdbbc-wdcg9"] Oct 31 00:15:22.075008 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.074985 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48"] Oct 31 00:15:22.077053 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:15:22.077026 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1970122b_758e_45f0_8d17_1f132f893b05.slice/crio-37edc692dba30b06b88021c5fabb2b10c4af7e77737562fc25e9e3de20c67196 WatchSource:0}: Error finding container 37edc692dba30b06b88021c5fabb2b10c4af7e77737562fc25e9e3de20c67196: Status 404 returned error can't find the container with id 37edc692dba30b06b88021c5fabb2b10c4af7e77737562fc25e9e3de20c67196 Oct 31 00:15:22.143230 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.143207 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt"] Oct 31 00:15:22.144613 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:15:22.144588 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbed3d8_69e3_4ab4_b309_8478c511571c.slice/crio-f3ed42a44e76a2c58ef2449589425f3e5b9a106a34d289155557c6e07a93dbf2 WatchSource:0}: Error finding container f3ed42a44e76a2c58ef2449589425f3e5b9a106a34d289155557c6e07a93dbf2: Status 404 returned error can't find the container with id f3ed42a44e76a2c58ef2449589425f3e5b9a106a34d289155557c6e07a93dbf2 Oct 31 00:15:22.146900 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.146868 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:15:22.146971 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.146934 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:15:22.146971 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.146965 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:15:22.601382 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.601351 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621bf2bc-d847-406e-b916-ed91c38a6fa1" path="/var/lib/kubelet/pods/621bf2bc-d847-406e-b916-ed91c38a6fa1/volumes" Oct 31 00:15:22.881105 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.881067 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" event={"ID":"3dbed3d8-69e3-4ab4-b309-8478c511571c","Type":"ContainerStarted","Data":"84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815"} Oct 31 00:15:22.881298 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.881124 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" event={"ID":"3dbed3d8-69e3-4ab4-b309-8478c511571c","Type":"ContainerStarted","Data":"f3ed42a44e76a2c58ef2449589425f3e5b9a106a34d289155557c6e07a93dbf2"} Oct 31 00:15:22.882561 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.882531 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" event={"ID":"1970122b-758e-45f0-8d17-1f132f893b05","Type":"ContainerStarted","Data":"b97c9bd850517c683cbfa34c4f6826930d213500af6d9c49e95761e5191cbf89"} Oct 31 00:15:22.882671 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.882576 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" event={"ID":"1970122b-758e-45f0-8d17-1f132f893b05","Type":"ContainerStarted","Data":"37edc692dba30b06b88021c5fabb2b10c4af7e77737562fc25e9e3de20c67196"} Oct 31 00:15:22.882671 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.882609 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:22.906391 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.906345 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" podStartSLOduration=1.9063311550000002 podStartE2EDuration="1.906331155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:22.904766834 +0000 UTC m=+3334.423767918" watchObservedRunningTime="2026-10-31 00:15:22.906331155 +0000 UTC m=+3334.425332244" Oct 31 00:15:22.925881 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:22.925839 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" podStartSLOduration=1.439944442 podStartE2EDuration="1.925826041s" totalImagesPullingTime="485.881599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:22.924085304 +0000 UTC m=+3334.443086394" watchObservedRunningTime="2026-10-31 00:15:22.925826041 +0000 UTC m=+3334.444827130" Oct 31 00:15:23.026986 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:23.026954 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx"] Oct 31 00:15:23.030846 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:23.030819 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bftqx"] Oct 31 00:15:23.838150 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:23.838119 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:23.843360 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:23.843338 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:23.888770 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:23.888742 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:23.889676 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:23.889658 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:24.602815 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:24.602785 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52006e8e-96f6-4cf0-8654-744bea2151d2" path="/var/lib/kubelet/pods/52006e8e-96f6-4cf0-8654-744bea2151d2/volumes" Oct 31 00:15:25.666005 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.665975 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz"] Oct 31 00:15:25.666431 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.666418 7570 cpu_manager.go:414] "containerMap: removing container" podUID="621bf2bc-d847-406e-b916-ed91c38a6fa1" containerName="maas-api" Oct 31 00:15:25.666596 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.666432 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621bf2bc-d847-406e-b916-ed91c38a6fa1" containerName="maas-api" Oct 31 00:15:25.666596 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.666551 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="621bf2bc-d847-406e-b916-ed91c38a6fa1" containerName="maas-api" Oct 31 00:15:25.670302 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.670281 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.674436 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.674414 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-48ebba5a-serving-cert\"" Oct 31 00:15:25.685154 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.685129 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz"] Oct 31 00:15:25.705782 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.705746 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-metrics-certs\") pod \"maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.705939 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.705820 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchqm\" (UniqueName: \"kubernetes.io/projected/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-kube-api-access-lchqm\") pod \"maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.705939 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.705916 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-openshift-service-ca\") pod \"maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.706063 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.705977 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-maas-api-tls\") pod \"maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.807050 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.807013 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-metrics-certs\") pod \"maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.807214 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.807057 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchqm\" (UniqueName: \"kubernetes.io/projected/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-kube-api-access-lchqm\") pod \"maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.807214 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.807103 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-openshift-service-ca\") pod \"maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.807214 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.807138 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-maas-api-tls\") pod \"maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.807822 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.807770 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-openshift-service-ca\") pod \"maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.809793 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.809771 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-maas-api-tls\") pod \"maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.809888 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.809832 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-metrics-certs\") pod \"maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.819530 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.819510 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchqm\" (UniqueName: \"kubernetes.io/projected/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-kube-api-access-lchqm\") pod \"maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:25.992774 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:25.992682 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:26.273919 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:26.273895 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz"] Oct 31 00:15:26.276066 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:15:26.276039 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bc0b20_e93c_480e_aaf8_efd6d3da7ce0.slice/crio-61d4e9b156011705b24d60e0179aac1aced6a21d72f03e23cd9d4a963e04b059 WatchSource:0}: Error finding container 61d4e9b156011705b24d60e0179aac1aced6a21d72f03e23cd9d4a963e04b059: Status 404 returned error can't find the container with id 61d4e9b156011705b24d60e0179aac1aced6a21d72f03e23cd9d4a963e04b059 Oct 31 00:15:26.911156 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:26.911123 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" event={"ID":"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0","Type":"ContainerStarted","Data":"69593ea364613c43af5c700e8025dcf94abcf0cc58f01eaac58431fd215d5db6"} Oct 31 00:15:26.911658 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:26.911184 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" event={"ID":"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0","Type":"ContainerStarted","Data":"61d4e9b156011705b24d60e0179aac1aced6a21d72f03e23cd9d4a963e04b059"} Oct 31 00:15:26.911658 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:26.911256 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:26.936356 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:26.936306 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" podStartSLOduration=1.462696501 podStartE2EDuration="1.936294116s" totalImagesPullingTime="473.597615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:26.933932571 +0000 UTC m=+3338.452933662" watchObservedRunningTime="2026-10-31 00:15:26.936294116 +0000 UTC m=+3338.455295205" Oct 31 00:15:28.913181 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:28.913153 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f5c9bcb67-2tr48" Oct 31 00:15:29.435593 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:15:29.435558 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:15:32.922990 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:32.922959 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:37.978398 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:37.978360 7570 generic.go:356] "Generic (PLEG): container finished" podID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerID="84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815" exitCode=0 Oct 31 00:15:37.978947 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:37.978441 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" event={"ID":"3dbed3d8-69e3-4ab4-b309-8478c511571c","Type":"ContainerDied","Data":"84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815"} Oct 31 00:15:37.978947 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:37.978930 7570 scope.go:122] "RemoveContainer" containerID="84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815" Oct 31 00:15:37.980187 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:37.980147 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:15:37.980300 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:37.980245 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:15:37.980300 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:37.980294 7570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:15:38.891487 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:38.889549 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:39.000487 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:38.985763 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" event={"ID":"3dbed3d8-69e3-4ab4-b309-8478c511571c","Type":"ContainerStarted","Data":"4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6"} Oct 31 00:15:39.838149 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:39.838113 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:39.839391 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:39.839358 7570 patch_prober.go:29] interesting pod/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 31 00:15:39.839549 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:39.839423 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 31 00:15:40.837840 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:40.837800 7570 patch_prober.go:29] interesting pod/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 31 00:15:40.838352 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:40.837856 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 31 00:15:41.837638 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:41.837605 7570 patch_prober.go:29] interesting pod/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 31 00:15:41.837813 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:41.837660 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 31 00:15:42.838392 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:42.838356 7570 patch_prober.go:29] interesting pod/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 31 00:15:42.838803 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:42.838414 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 31 00:15:43.838065 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:43.837976 7570 patch_prober.go:29] interesting pod/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 31 00:15:43.838065 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:43.838030 7570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 31 00:15:44.841703 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:44.841673 7570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:44.842134 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:44.841949 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:44.842735 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:44.842718 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:15:50.850162 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:50.850122 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz"] Oct 31 00:15:50.850741 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:50.850489 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" podUID="e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0" containerName="maas-api" containerID="cri-o://69593ea364613c43af5c700e8025dcf94abcf0cc58f01eaac58431fd215d5db6" gracePeriod=30 Oct 31 00:15:51.082803 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.082767 7570 generic.go:356] "Generic (PLEG): container finished" podID="e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0" containerID="69593ea364613c43af5c700e8025dcf94abcf0cc58f01eaac58431fd215d5db6" exitCode=0 Oct 31 00:15:51.082958 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.082840 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" event={"ID":"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0","Type":"ContainerDied","Data":"69593ea364613c43af5c700e8025dcf94abcf0cc58f01eaac58431fd215d5db6"} Oct 31 00:15:51.126080 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.126054 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:51.225973 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.225942 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-metrics-certs\") pod \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " Oct 31 00:15:51.226133 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.226008 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-maas-api-tls\") pod \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " Oct 31 00:15:51.226133 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.226088 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-openshift-service-ca\") pod \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " Oct 31 00:15:51.226215 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.226141 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-kube-api-access-lchqm\" (UniqueName: \"kubernetes.io/projected/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-kube-api-access-lchqm\") pod \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\" (UID: \"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0\") " Oct 31 00:15:51.226548 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.226517 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-openshift-service-ca" pod "e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0" (UID: "e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:51.226691 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.226627 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:51.228407 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.228379 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-kube-api-access-lchqm" pod "e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0" (UID: "e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0"). InnerVolumeSpecName "kube-api-access-lchqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:51.228536 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.228409 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-maas-api-tls" pod "e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0" (UID: "e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:15:51.228536 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.228440 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-metrics-certs" pod "e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0" (UID: "e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:15:51.327824 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.327784 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lchqm\" (UniqueName: \"kubernetes.io/projected/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-kube-api-access-lchqm\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:51.327824 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.327818 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:51.328033 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:51.327834 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:52.091299 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:52.091261 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" Oct 31 00:15:52.091822 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:52.091265 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz" event={"ID":"e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0","Type":"ContainerDied","Data":"61d4e9b156011705b24d60e0179aac1aced6a21d72f03e23cd9d4a963e04b059"} Oct 31 00:15:52.091822 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:52.091427 7570 scope.go:122] "RemoveContainer" containerID="69593ea364613c43af5c700e8025dcf94abcf0cc58f01eaac58431fd215d5db6" Oct 31 00:15:52.121215 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:52.121190 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz"] Oct 31 00:15:52.124606 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:52.124585 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-48ebba5a-77f655968f-lh4rz"] Oct 31 00:15:52.600894 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:52.600863 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0" path="/var/lib/kubelet/pods/e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0/volumes" Oct 31 00:15:56.662501 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:56.662439 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt"] Oct 31 00:15:56.663208 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:56.663176 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" containerID="cri-o://4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6" gracePeriod=30 Oct 31 00:15:59.411213 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:15:59.411173 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:15:59.844782 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:59.844751 7570 patch_prober.go:29] interesting pod/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:15:59.844933 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:15:59.844804 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:16:01.928406 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:01.928346 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:16:02.038057 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.038029 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-socket\") pod \"3dbed3d8-69e3-4ab4-b309-8478c511571c\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " Oct 31 00:16:02.038236 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.038070 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-certs\") pod \"3dbed3d8-69e3-4ab4-b309-8478c511571c\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " Oct 31 00:16:02.038236 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.038097 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-token\" (UniqueName: \"kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-token\") pod \"3dbed3d8-69e3-4ab4-b309-8478c511571c\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " Oct 31 00:16:02.038236 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.038123 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-credential-socket\") pod \"3dbed3d8-69e3-4ab4-b309-8478c511571c\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " Oct 31 00:16:02.038398 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.038307 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-podinfo\") pod \"3dbed3d8-69e3-4ab4-b309-8478c511571c\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " Oct 31 00:16:02.038398 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.038359 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-kube-api-access-vjvlg\" (UniqueName: \"kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-kube-api-access-vjvlg\") pod \"3dbed3d8-69e3-4ab4-b309-8478c511571c\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " Oct 31 00:16:02.038535 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.038349 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-socket" pod "3dbed3d8-69e3-4ab4-b309-8478c511571c" (UID: "3dbed3d8-69e3-4ab4-b309-8478c511571c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:02.038535 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.038441 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-certs" pod "3dbed3d8-69e3-4ab4-b309-8478c511571c" (UID: "3dbed3d8-69e3-4ab4-b309-8478c511571c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:02.038535 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.038481 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-credential-socket" pod "3dbed3d8-69e3-4ab4-b309-8478c511571c" (UID: "3dbed3d8-69e3-4ab4-b309-8478c511571c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:02.038535 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.038491 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-envoy\") pod \"3dbed3d8-69e3-4ab4-b309-8478c511571c\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " Oct 31 00:16:02.038744 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.038570 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbed3d8-69e3-4ab4-b309-8478c511571c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbed3d8-69e3-4ab4-b309-8478c511571c-istiod-ca-cert\") pod \"3dbed3d8-69e3-4ab4-b309-8478c511571c\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " Oct 31 00:16:02.038744 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.038627 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-data\") pod \"3dbed3d8-69e3-4ab4-b309-8478c511571c\" (UID: \"3dbed3d8-69e3-4ab4-b309-8478c511571c\") " Oct 31 00:16:02.039025 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.039004 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-data" pod "3dbed3d8-69e3-4ab4-b309-8478c511571c" (UID: "3dbed3d8-69e3-4ab4-b309-8478c511571c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:02.039137 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.039034 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbed3d8-69e3-4ab4-b309-8478c511571c-istiod-ca-cert" pod "3dbed3d8-69e3-4ab4-b309-8478c511571c" (UID: "3dbed3d8-69e3-4ab4-b309-8478c511571c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:02.039137 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.039106 7570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-data\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:02.039137 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.039126 7570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:02.039269 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.039140 7570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-workload-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:02.039269 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.039153 7570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-credential-socket\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:02.041199 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.041175 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-kube-api-access-vjvlg" pod "3dbed3d8-69e3-4ab4-b309-8478c511571c" (UID: "3dbed3d8-69e3-4ab4-b309-8478c511571c"). InnerVolumeSpecName "kube-api-access-vjvlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:02.041282 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.041200 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-envoy" pod "3dbed3d8-69e3-4ab4-b309-8478c511571c" (UID: "3dbed3d8-69e3-4ab4-b309-8478c511571c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:02.041282 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.041211 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-token" pod "3dbed3d8-69e3-4ab4-b309-8478c511571c" (UID: "3dbed3d8-69e3-4ab4-b309-8478c511571c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:02.041282 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.041251 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-podinfo" pod "3dbed3d8-69e3-4ab4-b309-8478c511571c" (UID: "3dbed3d8-69e3-4ab4-b309-8478c511571c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:16:02.140499 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.140451 7570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-podinfo\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:02.140499 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.140498 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvlg\" (UniqueName: \"kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-kube-api-access-vjvlg\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:02.140668 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.140510 7570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-envoy\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:02.140668 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.140519 7570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbed3d8-69e3-4ab4-b309-8478c511571c-istiod-ca-cert\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:02.140668 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.140527 7570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbed3d8-69e3-4ab4-b309-8478c511571c-istio-token\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:02.148284 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.148254 7570 generic.go:356] "Generic (PLEG): container finished" podID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerID="4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6" exitCode=0 Oct 31 00:16:02.148431 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.148300 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" Oct 31 00:16:02.148431 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.148318 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" event={"ID":"3dbed3d8-69e3-4ab4-b309-8478c511571c","Type":"ContainerDied","Data":"4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6"} Oct 31 00:16:02.148431 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.148374 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt" event={"ID":"3dbed3d8-69e3-4ab4-b309-8478c511571c","Type":"ContainerDied","Data":"f3ed42a44e76a2c58ef2449589425f3e5b9a106a34d289155557c6e07a93dbf2"} Oct 31 00:16:02.148431 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.148405 7570 scope.go:122] "RemoveContainer" containerID="4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6" Oct 31 00:16:02.160254 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.160235 7570 scope.go:122] "RemoveContainer" containerID="84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815" Oct 31 00:16:02.172939 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.172921 7570 scope.go:122] "RemoveContainer" containerID="4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6" Oct 31 00:16:02.173181 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:16:02.173160 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6\": container with ID starting with 4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6 not found: ID does not exist" containerID="4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6" Oct 31 00:16:02.173228 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.173196 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6"} err="failed to get container status \"4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6\": rpc error: code = NotFound desc = could not find container \"4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6\": container with ID starting with 4ce72cda84811a75d87e4ece2e9865f6404d32d69473e9726e4087bf306f36c6 not found: ID does not exist" Oct 31 00:16:02.173228 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.173215 7570 scope.go:122] "RemoveContainer" containerID="84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815" Oct 31 00:16:02.173488 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:16:02.173446 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815\": container with ID starting with 84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815 not found: ID does not exist" containerID="84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815" Oct 31 00:16:02.173541 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.173515 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815"} err="failed to get container status \"84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815\": rpc error: code = NotFound desc = could not find container \"84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815\": container with ID starting with 84311d4840c2fb82633f6a2905de3f3f795ed9e1701d1b5aef9b0ca61d17d815 not found: ID does not exist" Oct 31 00:16:02.182085 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.182029 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt"] Oct 31 00:16:02.187371 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.187348 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-48ebba5a-openshift-default-5467854bb5-qdbvt"] Oct 31 00:16:02.601590 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:02.601560 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" path="/var/lib/kubelet/pods/3dbed3d8-69e3-4ab4-b309-8478c511571c/volumes" Oct 31 00:16:03.059997 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.059911 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv"] Oct 31 00:16:03.061256 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.061230 7570 cpu_manager.go:414] "containerMap: removing container" podUID="e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0" containerName="maas-api" Oct 31 00:16:03.061256 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.061254 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0" containerName="maas-api" Oct 31 00:16:03.061452 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.061278 7570 cpu_manager.go:414] "containerMap: removing container" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" Oct 31 00:16:03.061452 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.061283 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" Oct 31 00:16:03.061452 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.061377 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" Oct 31 00:16:03.061452 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.061395 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2bc0b20-e93c-480e-aaf8-efd6d3da7ce0" containerName="maas-api" Oct 31 00:16:03.066526 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.066507 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.075119 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.075092 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-b33f34-serving-cert\"" Oct 31 00:16:03.083543 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.083517 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv"] Oct 31 00:16:03.149749 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.149718 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.149926 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.149875 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cec008-c166-4b08-ad7c-16137ceca94c-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.150015 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.149987 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl6w4\" (UniqueName: \"kubernetes.io/projected/b0cec008-c166-4b08-ad7c-16137ceca94c-kube-api-access-rl6w4\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.150063 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.150036 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-metrics-certs\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.183027 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.182997 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz"] Oct 31 00:16:03.183507 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.183493 7570 cpu_manager.go:414] "containerMap: removing container" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" Oct 31 00:16:03.183559 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.183508 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" Oct 31 00:16:03.183616 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.183607 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbed3d8-69e3-4ab4-b309-8478c511571c" containerName="istio-proxy" Oct 31 00:16:03.186645 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.186623 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" Oct 31 00:16:03.193155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.193125 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-b33f34-dockercfg-qpl2g\"" Oct 31 00:16:03.193155 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.193133 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-b33f34\"" Oct 31 00:16:03.200987 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.200957 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz"] Oct 31 00:16:03.214134 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.214106 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z"] Oct 31 00:16:03.218319 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.218296 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" Oct 31 00:16:03.227848 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.227822 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z"] Oct 31 00:16:03.251247 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.251222 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl6w4\" (UniqueName: \"kubernetes.io/projected/b0cec008-c166-4b08-ad7c-16137ceca94c-kube-api-access-rl6w4\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.251410 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.251264 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz\" (UID: \"59e543eb-c2cb-4ad8-a8bc-ce5335166db4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" Oct 31 00:16:03.251410 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.251303 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-metrics-certs\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.251565 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.251526 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.251669 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:16:03.251650 7570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-serving-cert: secret "maas-api-e2e-authcfg-http-b33f34-serving-cert" not found Oct 31 00:16:03.251722 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.251667 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6k5\" (UniqueName: \"kubernetes.io/projected/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-kube-api-access-xc6k5\") pod \"payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz\" (UID: \"59e543eb-c2cb-4ad8-a8bc-ce5335166db4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" Oct 31 00:16:03.251771 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:16:03.251723 7570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-maas-api-tls podName:b0cec008-c166-4b08-ad7c-16137ceca94c nodeName:}" failed. No retries permitted until 2026-10-31 00:16:03.751699518 +0000 UTC m=+3375.270700586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-maas-api-tls") pod "maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" (UID: "b0cec008-c166-4b08-ad7c-16137ceca94c") : secret "maas-api-e2e-authcfg-http-b33f34-serving-cert" not found Oct 31 00:16:03.251771 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.251766 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cec008-c166-4b08-ad7c-16137ceca94c-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.252326 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.252300 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cec008-c166-4b08-ad7c-16137ceca94c-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.253832 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.253812 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-metrics-certs\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.270176 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.270151 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl6w4\" (UniqueName: \"kubernetes.io/projected/b0cec008-c166-4b08-ad7c-16137ceca94c-kube-api-access-rl6w4\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.352996 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.352965 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6k5\" (UniqueName: \"kubernetes.io/projected/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-kube-api-access-xc6k5\") pod \"payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz\" (UID: \"59e543eb-c2cb-4ad8-a8bc-ce5335166db4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" Oct 31 00:16:03.353189 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.353014 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qbv\" (UniqueName: \"kubernetes.io/projected/6b3d9646-c964-444b-83c9-59b8adf486db-kube-api-access-z6qbv\") pod \"payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z\" (UID: \"6b3d9646-c964-444b-83c9-59b8adf486db\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" Oct 31 00:16:03.353189 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.353066 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz\" (UID: \"59e543eb-c2cb-4ad8-a8bc-ce5335166db4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" Oct 31 00:16:03.353287 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.353196 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d9646-c964-444b-83c9-59b8adf486db-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z\" (UID: \"6b3d9646-c964-444b-83c9-59b8adf486db\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" Oct 31 00:16:03.353842 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.353824 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz\" (UID: \"59e543eb-c2cb-4ad8-a8bc-ce5335166db4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" Oct 31 00:16:03.366861 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.366838 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6k5\" (UniqueName: \"kubernetes.io/projected/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-kube-api-access-xc6k5\") pod \"payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz\" (UID: \"59e543eb-c2cb-4ad8-a8bc-ce5335166db4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" Oct 31 00:16:03.454732 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.454703 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qbv\" (UniqueName: \"kubernetes.io/projected/6b3d9646-c964-444b-83c9-59b8adf486db-kube-api-access-z6qbv\") pod \"payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z\" (UID: \"6b3d9646-c964-444b-83c9-59b8adf486db\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" Oct 31 00:16:03.454876 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.454825 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d9646-c964-444b-83c9-59b8adf486db-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z\" (UID: \"6b3d9646-c964-444b-83c9-59b8adf486db\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" Oct 31 00:16:03.455493 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.455477 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d9646-c964-444b-83c9-59b8adf486db-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z\" (UID: \"6b3d9646-c964-444b-83c9-59b8adf486db\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" Oct 31 00:16:03.467645 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.467624 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qbv\" (UniqueName: \"kubernetes.io/projected/6b3d9646-c964-444b-83c9-59b8adf486db-kube-api-access-z6qbv\") pod \"payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z\" (UID: \"6b3d9646-c964-444b-83c9-59b8adf486db\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" Oct 31 00:16:03.503656 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.503631 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" Oct 31 00:16:03.537153 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.537125 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" Oct 31 00:16:03.757759 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.757725 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.760400 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.760374 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:03.782870 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.782845 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz"] Oct 31 00:16:03.785680 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:16:03.785649 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e543eb_c2cb_4ad8_a8bc_ce5335166db4.slice/crio-10bfe508afde594f4ed584a4573a7883de2368de3fbac7c0171b8161f69c6b44 WatchSource:0}: Error finding container 10bfe508afde594f4ed584a4573a7883de2368de3fbac7c0171b8161f69c6b44: Status 404 returned error can't find the container with id 10bfe508afde594f4ed584a4573a7883de2368de3fbac7c0171b8161f69c6b44 Oct 31 00:16:03.829545 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.829522 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z"] Oct 31 00:16:03.832582 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:16:03.832559 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3d9646_c964_444b_83c9_59b8adf486db.slice/crio-c53d380d5f9d055aabc2f94bd0912bbdc8a61c3d505df14f57310b322b004937 WatchSource:0}: Error finding container c53d380d5f9d055aabc2f94bd0912bbdc8a61c3d505df14f57310b322b004937: Status 404 returned error can't find the container with id c53d380d5f9d055aabc2f94bd0912bbdc8a61c3d505df14f57310b322b004937 Oct 31 00:16:03.984815 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:03.984784 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:04.164924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:04.164840 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" event={"ID":"6b3d9646-c964-444b-83c9-59b8adf486db","Type":"ContainerStarted","Data":"a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9"} Oct 31 00:16:04.164924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:04.164908 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" event={"ID":"6b3d9646-c964-444b-83c9-59b8adf486db","Type":"ContainerStarted","Data":"c53d380d5f9d055aabc2f94bd0912bbdc8a61c3d505df14f57310b322b004937"} Oct 31 00:16:04.165405 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:04.165078 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" Oct 31 00:16:04.166609 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:04.166586 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" event={"ID":"59e543eb-c2cb-4ad8-a8bc-ce5335166db4","Type":"ContainerStarted","Data":"d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12"} Oct 31 00:16:04.166745 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:04.166627 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" Oct 31 00:16:04.166745 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:04.166643 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" event={"ID":"59e543eb-c2cb-4ad8-a8bc-ce5335166db4","Type":"ContainerStarted","Data":"10bfe508afde594f4ed584a4573a7883de2368de3fbac7c0171b8161f69c6b44"} Oct 31 00:16:04.183991 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:04.183601 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" podStartSLOduration=1.183585909 podStartE2EDuration="1.183585909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:04.183189644 +0000 UTC m=+3375.702190733" watchObservedRunningTime="2026-10-31 00:16:04.183585909 +0000 UTC m=+3375.702586997" Oct 31 00:16:04.201483 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:04.201419 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" podStartSLOduration=1.201408116 podStartE2EDuration="1.201408116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:04.2008113 +0000 UTC m=+3375.719812390" watchObservedRunningTime="2026-10-31 00:16:04.201408116 +0000 UTC m=+3375.720409200" Oct 31 00:16:04.265885 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:04.265820 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv"] Oct 31 00:16:04.269128 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:16:04.269098 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cec008_c166_4b08_ad7c_16137ceca94c.slice/crio-2545b515e7dac7a2764792e89fc2b5c92b74453fcadf11a601f863bbafdc04e8 WatchSource:0}: Error finding container 2545b515e7dac7a2764792e89fc2b5c92b74453fcadf11a601f863bbafdc04e8: Status 404 returned error can't find the container with id 2545b515e7dac7a2764792e89fc2b5c92b74453fcadf11a601f863bbafdc04e8 Oct 31 00:16:05.174871 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:05.174828 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" event={"ID":"b0cec008-c166-4b08-ad7c-16137ceca94c","Type":"ContainerStarted","Data":"a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a"} Oct 31 00:16:05.175403 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:05.174892 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" event={"ID":"b0cec008-c166-4b08-ad7c-16137ceca94c","Type":"ContainerStarted","Data":"2545b515e7dac7a2764792e89fc2b5c92b74453fcadf11a601f863bbafdc04e8"} Oct 31 00:16:05.175403 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:05.174922 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:05.196827 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:05.196771 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" podStartSLOduration=1.6474205039999998 podStartE2EDuration="2.19675673s" totalImagesPullingTime="549.336226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:05.19576236 +0000 UTC m=+3376.714763463" watchObservedRunningTime="2026-10-31 00:16:05.19675673 +0000 UTC m=+3376.715757819" Oct 31 00:16:11.187016 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:11.186986 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:16:15.178367 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:15.178335 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" Oct 31 00:16:15.178827 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:15.178403 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" Oct 31 00:16:29.412504 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:16:29.412441 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:16:44.760881 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:44.760850 7570 scope.go:122] "RemoveContainer" containerID="b13cab2b43cda70aefa546bff4f75c9e6c290add6144b47df16b99878029ce04" Oct 31 00:16:44.772911 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:44.772892 7570 scope.go:122] "RemoveContainer" containerID="298941680c17d5c656420abb128b82d45868dd8617579f6851dfff31d8ae7946" Oct 31 00:16:44.781403 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:16:44.781379 7570 scope.go:122] "RemoveContainer" containerID="f66237f71959c0002a8ba94a573420a2d77c632c671b34dfcb55d7e41c4230ef" Oct 31 00:16:59.412853 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:16:59.412815 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:17:02.023406 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:02.023364 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274"] Oct 31 00:17:02.028915 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:02.027811 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4f274"] Oct 31 00:17:02.601265 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:02.601231 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f346c4-b2c1-433a-82f3-d9de686c6e40" path="/var/lib/kubelet/pods/49f346c4-b2c1-433a-82f3-d9de686c6e40/volumes" Oct 31 00:17:21.035577 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:21.035490 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk"] Oct 31 00:17:21.037797 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:21.037775 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jdjqk"] Oct 31 00:17:22.601157 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:22.601130 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d971e795-94ee-4fbc-809a-9d605c2e7b29" path="/var/lib/kubelet/pods/d971e795-94ee-4fbc-809a-9d605c2e7b29/volumes" Oct 31 00:17:29.413245 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:17:29.413204 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:17:43.662300 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:43.662273 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log" Oct 31 00:17:45.660227 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:45.660201 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsv62_29c3d1e4-09b5-4712-ad6d-9a32434def2d/csi-driver/0.log" Oct 31 00:17:47.434278 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:47.434238 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z"] Oct 31 00:17:47.456963 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:47.456864 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz"] Oct 31 00:17:52.435144 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.435096 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" podUID="6b3d9646-c964-444b-83c9-59b8adf486db" containerName="payload-processing" containerID="cri-o://a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9" gracePeriod=25 Oct 31 00:17:52.458187 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.458151 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" podUID="59e543eb-c2cb-4ad8-a8bc-ce5335166db4" containerName="payload-pre-processing" containerID="cri-o://d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12" gracePeriod=25 Oct 31 00:17:52.731051 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.731020 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" Oct 31 00:17:52.762698 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.762671 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" Oct 31 00:17:52.794674 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.794633 7570 generic.go:356] "Generic (PLEG): container finished" podID="59e543eb-c2cb-4ad8-a8bc-ce5335166db4" containerID="d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12" exitCode=0 Oct 31 00:17:52.794841 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.794704 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" Oct 31 00:17:52.794841 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.794714 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" event={"ID":"59e543eb-c2cb-4ad8-a8bc-ce5335166db4","Type":"ContainerDied","Data":"d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12"} Oct 31 00:17:52.794841 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.794770 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz" event={"ID":"59e543eb-c2cb-4ad8-a8bc-ce5335166db4","Type":"ContainerDied","Data":"10bfe508afde594f4ed584a4573a7883de2368de3fbac7c0171b8161f69c6b44"} Oct 31 00:17:52.794841 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.794800 7570 scope.go:122] "RemoveContainer" containerID="d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12" Oct 31 00:17:52.796748 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.796660 7570 generic.go:356] "Generic (PLEG): container finished" podID="6b3d9646-c964-444b-83c9-59b8adf486db" containerID="a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9" exitCode=0 Oct 31 00:17:52.796748 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.796717 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" event={"ID":"6b3d9646-c964-444b-83c9-59b8adf486db","Type":"ContainerDied","Data":"a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9"} Oct 31 00:17:52.796748 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.796751 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" event={"ID":"6b3d9646-c964-444b-83c9-59b8adf486db","Type":"ContainerDied","Data":"c53d380d5f9d055aabc2f94bd0912bbdc8a61c3d505df14f57310b322b004937"} Oct 31 00:17:52.797011 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.796802 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z" Oct 31 00:17:52.803804 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.803778 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-plugins-config-volume\") pod \"59e543eb-c2cb-4ad8-a8bc-ce5335166db4\" (UID: \"59e543eb-c2cb-4ad8-a8bc-ce5335166db4\") " Oct 31 00:17:52.804123 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.804103 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-kube-api-access-xc6k5\" (UniqueName: \"kubernetes.io/projected/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-kube-api-access-xc6k5\") pod \"59e543eb-c2cb-4ad8-a8bc-ce5335166db4\" (UID: \"59e543eb-c2cb-4ad8-a8bc-ce5335166db4\") " Oct 31 00:17:52.804277 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.804261 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3d9646-c964-444b-83c9-59b8adf486db-kube-api-access-z6qbv\" (UniqueName: \"kubernetes.io/projected/6b3d9646-c964-444b-83c9-59b8adf486db-kube-api-access-z6qbv\") pod \"6b3d9646-c964-444b-83c9-59b8adf486db\" (UID: \"6b3d9646-c964-444b-83c9-59b8adf486db\") " Oct 31 00:17:52.804408 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.804384 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3d9646-c964-444b-83c9-59b8adf486db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d9646-c964-444b-83c9-59b8adf486db-plugins-config-volume\") pod \"6b3d9646-c964-444b-83c9-59b8adf486db\" (UID: \"6b3d9646-c964-444b-83c9-59b8adf486db\") " Oct 31 00:17:52.804604 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.804121 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-plugins-config-volume" pod "59e543eb-c2cb-4ad8-a8bc-ce5335166db4" (UID: "59e543eb-c2cb-4ad8-a8bc-ce5335166db4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:52.804858 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.804841 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:52.804962 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.804935 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3d9646-c964-444b-83c9-59b8adf486db-plugins-config-volume" pod "6b3d9646-c964-444b-83c9-59b8adf486db" (UID: "6b3d9646-c964-444b-83c9-59b8adf486db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:52.806089 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.806072 7570 scope.go:122] "RemoveContainer" containerID="d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12" Oct 31 00:17:52.806821 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:17:52.806677 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12\": container with ID starting with d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12 not found: ID does not exist" containerID="d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12" Oct 31 00:17:52.806821 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.806712 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12"} err="failed to get container status \"d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12\": rpc error: code = NotFound desc = could not find container \"d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12\": container with ID starting with d8caea28791df59ceff88cf2554c93535dae43b521454da0f462be7c41fd5a12 not found: ID does not exist" Oct 31 00:17:52.806821 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.806735 7570 scope.go:122] "RemoveContainer" containerID="a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9" Oct 31 00:17:52.807146 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.807122 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3d9646-c964-444b-83c9-59b8adf486db-kube-api-access-z6qbv" pod "6b3d9646-c964-444b-83c9-59b8adf486db" (UID: "6b3d9646-c964-444b-83c9-59b8adf486db"). InnerVolumeSpecName "kube-api-access-z6qbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:52.808426 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.808401 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-kube-api-access-xc6k5" pod "59e543eb-c2cb-4ad8-a8bc-ce5335166db4" (UID: "59e543eb-c2cb-4ad8-a8bc-ce5335166db4"). InnerVolumeSpecName "kube-api-access-xc6k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:52.815621 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.815593 7570 scope.go:122] "RemoveContainer" containerID="a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9" Oct 31 00:17:52.815982 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:17:52.815958 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9\": container with ID starting with a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9 not found: ID does not exist" containerID="a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9" Oct 31 00:17:52.816071 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.815989 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9"} err="failed to get container status \"a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9\": rpc error: code = NotFound desc = could not find container \"a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9\": container with ID starting with a6984913b9bc62f951d1f160d0ec38c0aac2511dcb85e8c87ecc3a15501a29d9 not found: ID does not exist" Oct 31 00:17:52.906163 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.906117 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc6k5\" (UniqueName: \"kubernetes.io/projected/59e543eb-c2cb-4ad8-a8bc-ce5335166db4-kube-api-access-xc6k5\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:52.906163 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.906158 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6qbv\" (UniqueName: \"kubernetes.io/projected/6b3d9646-c964-444b-83c9-59b8adf486db-kube-api-access-z6qbv\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:52.906163 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.906173 7570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d9646-c964-444b-83c9-59b8adf486db-plugins-config-volume\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:52.916306 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.916261 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv"] Oct 31 00:17:52.916566 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:52.916536 7570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" podUID="b0cec008-c166-4b08-ad7c-16137ceca94c" containerName="maas-api" containerID="cri-o://a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a" gracePeriod=30 Oct 31 00:17:53.123440 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.123343 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz"] Oct 31 00:17:53.127597 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.127570 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b33f34-69b9678c6c-j5mbz"] Oct 31 00:17:53.138826 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.138800 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z"] Oct 31 00:17:53.142756 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.142725 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b33f34-86c64bcc8f-fg49z"] Oct 31 00:17:53.164768 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.164743 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:17:53.209392 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.209355 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cec008-c166-4b08-ad7c-16137ceca94c-kube-api-access-rl6w4\" (UniqueName: \"kubernetes.io/projected/b0cec008-c166-4b08-ad7c-16137ceca94c-kube-api-access-rl6w4\") pod \"b0cec008-c166-4b08-ad7c-16137ceca94c\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " Oct 31 00:17:53.209724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.209532 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-maas-api-tls\") pod \"b0cec008-c166-4b08-ad7c-16137ceca94c\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " Oct 31 00:17:53.209724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.209588 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-metrics-certs\") pod \"b0cec008-c166-4b08-ad7c-16137ceca94c\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " Oct 31 00:17:53.209724 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.209625 7570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cec008-c166-4b08-ad7c-16137ceca94c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cec008-c166-4b08-ad7c-16137ceca94c-openshift-service-ca\") pod \"b0cec008-c166-4b08-ad7c-16137ceca94c\" (UID: \"b0cec008-c166-4b08-ad7c-16137ceca94c\") " Oct 31 00:17:53.210512 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.210426 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cec008-c166-4b08-ad7c-16137ceca94c-openshift-service-ca" pod "b0cec008-c166-4b08-ad7c-16137ceca94c" (UID: "b0cec008-c166-4b08-ad7c-16137ceca94c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:53.212418 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.212389 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cec008-c166-4b08-ad7c-16137ceca94c-kube-api-access-rl6w4" pod "b0cec008-c166-4b08-ad7c-16137ceca94c" (UID: "b0cec008-c166-4b08-ad7c-16137ceca94c"). InnerVolumeSpecName "kube-api-access-rl6w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:53.212549 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.212489 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-metrics-certs" pod "b0cec008-c166-4b08-ad7c-16137ceca94c" (UID: "b0cec008-c166-4b08-ad7c-16137ceca94c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:53.213587 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.213556 7570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-maas-api-tls" pod "b0cec008-c166-4b08-ad7c-16137ceca94c" (UID: "b0cec008-c166-4b08-ad7c-16137ceca94c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:53.311365 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.311270 7570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-maas-api-tls\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:53.311365 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.311311 7570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cec008-c166-4b08-ad7c-16137ceca94c-metrics-certs\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:53.311365 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.311322 7570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cec008-c166-4b08-ad7c-16137ceca94c-openshift-service-ca\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:53.311365 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.311333 7570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl6w4\" (UniqueName: \"kubernetes.io/projected/b0cec008-c166-4b08-ad7c-16137ceca94c-kube-api-access-rl6w4\") on node \"ip-10-0-137-81.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:53.806251 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.806216 7570 generic.go:356] "Generic (PLEG): container finished" podID="b0cec008-c166-4b08-ad7c-16137ceca94c" containerID="a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a" exitCode=0 Oct 31 00:17:53.806773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.806262 7570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" Oct 31 00:17:53.806773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.806280 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" event={"ID":"b0cec008-c166-4b08-ad7c-16137ceca94c","Type":"ContainerDied","Data":"a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a"} Oct 31 00:17:53.806773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.806334 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv" event={"ID":"b0cec008-c166-4b08-ad7c-16137ceca94c","Type":"ContainerDied","Data":"2545b515e7dac7a2764792e89fc2b5c92b74453fcadf11a601f863bbafdc04e8"} Oct 31 00:17:53.806773 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.806364 7570 scope.go:122] "RemoveContainer" containerID="a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a" Oct 31 00:17:53.816558 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.816541 7570 scope.go:122] "RemoveContainer" containerID="a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a" Oct 31 00:17:53.816819 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:17:53.816798 7570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a\": container with ID starting with a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a not found: ID does not exist" containerID="a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a" Oct 31 00:17:53.816861 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.816828 7570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a"} err="failed to get container status \"a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a\": rpc error: code = NotFound desc = could not find container \"a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a\": container with ID starting with a5639319104431769bdf7016f39f2ac69aefc32524191aaa409b5d1fec91948a not found: ID does not exist" Oct 31 00:17:53.829586 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.829561 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv"] Oct 31 00:17:53.833600 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:53.833578 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b33f34-6c46bb844f-rbzcv"] Oct 31 00:17:54.602697 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:54.602659 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e543eb-c2cb-4ad8-a8bc-ce5335166db4" path="/var/lib/kubelet/pods/59e543eb-c2cb-4ad8-a8bc-ce5335166db4/volumes" Oct 31 00:17:54.603116 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:54.603102 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3d9646-c964-444b-83c9-59b8adf486db" path="/var/lib/kubelet/pods/6b3d9646-c964-444b-83c9-59b8adf486db/volumes" Oct 31 00:17:54.603531 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:17:54.603518 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cec008-c166-4b08-ad7c-16137ceca94c" path="/var/lib/kubelet/pods/b0cec008-c166-4b08-ad7c-16137ceca94c/volumes" Oct 31 00:17:59.411506 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:17:59.411450 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:18:25.036851 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:18:25.036799 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj"] Oct 31 00:18:25.040548 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:18:25.040518 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6fafe5-jwlwj"] Oct 31 00:18:26.619621 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:18:26.619586 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9c9461-8de7-44f7-bbff-4df18afe7bfc" path="/var/lib/kubelet/pods/0f9c9461-8de7-44f7-bbff-4df18afe7bfc/volumes" Oct 31 00:18:29.411770 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:18:29.411736 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:18:44.827513 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:18:44.827410 7570 scope.go:122] "RemoveContainer" containerID="bf1fe8d0dd76fbdff824c2f1ee0908cbaa5b29068eda0fe609b37543721b09a9" Oct 31 00:18:44.836339 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:18:44.836319 7570 scope.go:122] "RemoveContainer" containerID="8d0b7303c7f92eed8b63eb107a2b72d08c1ca91240b225e7c186e55ef1fdac4a" Oct 31 00:18:49.545408 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:18:49.545378 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74dc7574c6-6rws8_3dc85af3-2ef5-4005-a8c5-d515a92d2eee/authorino/0.log" Oct 31 00:18:50.941485 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:18:50.940582 7570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-g4dbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:50.941485 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:18:50.940703 7570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-g4dbp" podUID="229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:59.411788 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:18:59.411745 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:19:00.974774 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:00.974742 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5n89j_20fc3e5f-362e-41f7-bb52-2b171d7af818/manager/1.log" Oct 31 00:19:01.851705 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:01.851672 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4blbf_1793bf9b-92db-4dae-bfd6-f86e5b076739/manager/1.log" Oct 31 00:19:02.498053 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:02.498016 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5n89j_20fc3e5f-362e-41f7-bb52-2b171d7af818/manager/0.log" Oct 31 00:19:02.712243 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:02.712190 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4blbf_1793bf9b-92db-4dae-bfd6-f86e5b076739/manager/0.log" Oct 31 00:19:03.290047 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:03.290014 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7f5c9bcb67-2tr48_1970122b-758e-45f0-8d17-1f132f893b05/maas-api/0.log" Oct 31 00:19:03.848948 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:03.848920 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-xdzx7_4bc96048-571a-42f1-89d1-e5c754d1ee63/revoke-keys/0.log" Oct 31 00:19:04.029578 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:04.029552 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-4dbhx_a1c40240-9d81-48f6-9b7c-f1f256d6a76b/revoke-keys/0.log" Oct 31 00:19:06.337103 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:06.337073 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74dc7574c6-6rws8_3dc85af3-2ef5-4005-a8c5-d515a92d2eee/authorino/0.log" Oct 31 00:19:08.990608 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:08.990574 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dfd47b86c-gjwxh_2b7be075-e8bb-4db1-bfab-782276c29b6e/kube-auth-proxy/0.log" Oct 31 00:19:09.185719 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:09.185674 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-7rp6c_e7406dbc-b7ad-4b0c-ac37-c85b302bf76e/istio-proxy/0.log" Oct 31 00:19:09.740902 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:09.740871 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-g4dbp_229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d/istio-proxy/0.log" Oct 31 00:19:10.502416 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:10.502385 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-p5kqz_9a41f90f-265b-4a47-a2bc-9239bbd1822b/istio-proxy/0.log" Oct 31 00:19:12.806569 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:12.806542 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-c6888_eeffad66-ea6b-486f-abd8-63e68120af4c/main/0.log" Oct 31 00:19:13.171429 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:13.171398 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-6pxqs_f87cf2ab-c80c-4680-9c10-497e2a11235b/main/0.log" Oct 31 00:19:13.359169 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:13.359138 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx_dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e/main/0.log" Oct 31 00:19:13.907532 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:13.907496 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2_5275c1e3-85ea-449f-b526-7c3cbb14ed4c/main/0.log" Oct 31 00:19:14.848650 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:14.848619 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74dc7574c6-6rws8_3dc85af3-2ef5-4005-a8c5-d515a92d2eee/authorino/0.log" Oct 31 00:19:24.715098 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:24.715061 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5n89j_20fc3e5f-362e-41f7-bb52-2b171d7af818/manager/1.log" Oct 31 00:19:25.571821 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:25.571792 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4blbf_1793bf9b-92db-4dae-bfd6-f86e5b076739/manager/1.log" Oct 31 00:19:26.354490 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:26.354447 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7f5c9bcb67-2tr48_1970122b-758e-45f0-8d17-1f132f893b05/maas-api/0.log" Oct 31 00:19:26.938062 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:26.938014 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-xdzx7_4bc96048-571a-42f1-89d1-e5c754d1ee63/revoke-keys/0.log" Oct 31 00:19:27.115441 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:27.115412 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-4dbhx_a1c40240-9d81-48f6-9b7c-f1f256d6a76b/revoke-keys/0.log" Oct 31 00:19:28.966516 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:28.966484 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74dc7574c6-6rws8_3dc85af3-2ef5-4005-a8c5-d515a92d2eee/authorino/0.log" Oct 31 00:19:29.414302 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:19:29.414268 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:19:31.022812 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:31.022785 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx"] Oct 31 00:19:31.025068 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:31.025043 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4dbhx"] Oct 31 00:19:31.354712 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:31.354690 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dfd47b86c-gjwxh_2b7be075-e8bb-4db1-bfab-782276c29b6e/kube-auth-proxy/0.log" Oct 31 00:19:31.541137 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:31.541105 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-7rp6c_e7406dbc-b7ad-4b0c-ac37-c85b302bf76e/istio-proxy/0.log" Oct 31 00:19:32.111357 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:32.111318 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-g4dbp_229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d/istio-proxy/0.log" Oct 31 00:19:32.601403 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:32.601363 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c40240-9d81-48f6-9b7c-f1f256d6a76b" path="/var/lib/kubelet/pods/a1c40240-9d81-48f6-9b7c-f1f256d6a76b/volumes" Oct 31 00:19:32.861673 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:32.861593 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-p5kqz_9a41f90f-265b-4a47-a2bc-9239bbd1822b/istio-proxy/0.log" Oct 31 00:19:34.754733 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:34.754701 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-c6888_eeffad66-ea6b-486f-abd8-63e68120af4c/main/0.log" Oct 31 00:19:35.112522 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:35.112455 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-6pxqs_f87cf2ab-c80c-4680-9c10-497e2a11235b/main/0.log" Oct 31 00:19:35.301535 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:35.301501 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88l2tx_dd6cf0ef-7ff3-4c02-8e20-bc39ac49bd9e/main/0.log" Oct 31 00:19:35.868477 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:35.868429 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-t67k2_5275c1e3-85ea-449f-b526-7c3cbb14ed4c/main/0.log" Oct 31 00:19:44.874428 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:44.874396 7570 scope.go:122] "RemoveContainer" containerID="246b77730e443a3e495de7ec4e3d51ec7dacabeb2d7bd5c3f2053006bc74e205" Oct 31 00:19:44.883675 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:44.883653 7570 scope.go:122] "RemoveContainer" containerID="f65e56feff1d0eb313fdd4063cf4b72b9a321279efcca2b3295604649b2e4083" Oct 31 00:19:45.024320 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:45.024278 7570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7"] Oct 31 00:19:45.026621 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:45.026599 7570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xdzx7"] Oct 31 00:19:46.601423 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:19:46.601390 7570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc96048-571a-42f1-89d1-e5c754d1ee63" path="/var/lib/kubelet/pods/4bc96048-571a-42f1-89d1-e5c754d1ee63/volumes" Oct 31 00:19:59.412195 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:19:59.412158 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:20:29.411563 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:20:29.411527 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:20:44.907718 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:20:44.907684 7570 scope.go:122] "RemoveContainer" containerID="ab38fb48c9a4a82bfb51779dea82429850efe28eb06b77762204de916ceda0d5" Oct 31 00:20:44.916836 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:20:44.916815 7570 scope.go:122] "RemoveContainer" containerID="2209d59a92739e25d1861b1e3ffe854b48956d2ac62c1255bb919d4f84a0680c" Oct 31 00:20:59.411161 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:20:59.411123 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:21:08.834276 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:08.834243 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6sfch_6ef6a2af-7976-41e3-af4e-c7e0cb5e6b3f/global-pull-secret-syncer/0.log" Oct 31 00:21:09.299171 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:09.299096 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-89shx_d3eeb220-e1ba-46e2-94c8-2bb19e02dde9/konnectivity-agent/0.log" Oct 31 00:21:09.418488 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:09.418438 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-81.us-east-2.compute.internal_70da0cbd0028ec04eeaec823ad5f3973/haproxy/0.log" Oct 31 00:21:15.912932 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:15.912903 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74dc7574c6-6rws8_3dc85af3-2ef5-4005-a8c5-d515a92d2eee/authorino/0.log" Oct 31 00:21:17.864110 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:17.864076 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/alertmanager/0.log" Oct 31 00:21:17.893432 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:17.893405 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/config-reloader/0.log" Oct 31 00:21:17.919695 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:17.919673 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/kube-rbac-proxy-web/0.log" Oct 31 00:21:17.943798 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:17.943778 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/kube-rbac-proxy/0.log" Oct 31 00:21:17.971101 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:17.971083 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/kube-rbac-proxy-metric/0.log" Oct 31 00:21:17.998510 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:17.998487 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/prom-label-proxy/0.log" Oct 31 00:21:18.025259 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.025239 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5973a913-5fc2-45b7-affa-4184ee2da449/init-config-reloader/0.log" Oct 31 00:21:18.108982 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.108958 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-thls4_3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/kube-state-metrics/0.log" Oct 31 00:21:18.136162 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.136103 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-thls4_3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/kube-rbac-proxy-main/0.log" Oct 31 00:21:18.166364 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.166345 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-thls4_3fd5ef23-9d7a-44c2-b0fe-653a2cba55e0/kube-rbac-proxy-self/0.log" Oct 31 00:21:18.205880 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.205857 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5946dc94c7-djbgv_ed360b42-1a02-4474-9bf1-2117d1b438f1/metrics-server/1.log" Oct 31 00:21:18.209256 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.209241 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5946dc94c7-djbgv_ed360b42-1a02-4474-9bf1-2117d1b438f1/metrics-server/0.log" Oct 31 00:21:18.229213 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.229193 7570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4"] Oct 31 00:21:18.229654 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.229641 7570 cpu_manager.go:414] "containerMap: removing container" podUID="6b3d9646-c964-444b-83c9-59b8adf486db" containerName="payload-processing" Oct 31 00:21:18.229699 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.229656 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3d9646-c964-444b-83c9-59b8adf486db" containerName="payload-processing" Oct 31 00:21:18.229699 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.229677 7570 cpu_manager.go:414] "containerMap: removing container" podUID="59e543eb-c2cb-4ad8-a8bc-ce5335166db4" containerName="payload-pre-processing" Oct 31 00:21:18.229699 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.229683 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e543eb-c2cb-4ad8-a8bc-ce5335166db4" containerName="payload-pre-processing" Oct 31 00:21:18.229790 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.229705 7570 cpu_manager.go:414] "containerMap: removing container" podUID="b0cec008-c166-4b08-ad7c-16137ceca94c" containerName="maas-api" Oct 31 00:21:18.229790 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.229710 7570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cec008-c166-4b08-ad7c-16137ceca94c" containerName="maas-api" Oct 31 00:21:18.229790 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.229777 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cec008-c166-4b08-ad7c-16137ceca94c" containerName="maas-api" Oct 31 00:21:18.229790 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.229790 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3d9646-c964-444b-83c9-59b8adf486db" containerName="payload-processing" Oct 31 00:21:18.229908 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.229799 7570 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e543eb-c2cb-4ad8-a8bc-ce5335166db4" containerName="payload-pre-processing" Oct 31 00:21:18.233238 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.233220 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.242057 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.242031 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4"] Oct 31 00:21:18.242554 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.242351 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrd6t\"/\"kube-root-ca.crt\"" Oct 31 00:21:18.242554 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.242377 7570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrd6t\"/\"openshift-service-ca.crt\"" Oct 31 00:21:18.242554 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.242428 7570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lrd6t\"/\"default-dockercfg-7c4gn\"" Oct 31 00:21:18.257053 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.257029 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-g86k2_9c20fb11-2dcf-445e-96cd-4d97045a2ab9/monitoring-plugin/0.log" Oct 31 00:21:18.294891 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.294868 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hq9sq_a0371e4c-ff4b-4867-98a1-bb598ab03845/node-exporter/0.log" Oct 31 00:21:18.319401 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.319381 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hq9sq_a0371e4c-ff4b-4867-98a1-bb598ab03845/kube-rbac-proxy/0.log" Oct 31 00:21:18.344341 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.344323 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hq9sq_a0371e4c-ff4b-4867-98a1-bb598ab03845/init-textfile/0.log" Oct 31 00:21:18.390517 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.390422 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be1ce9de-0f29-4d25-9039-00546ba0d386-lib-modules\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.390517 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.390504 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be1ce9de-0f29-4d25-9039-00546ba0d386-proc\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.390658 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.390595 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be1ce9de-0f29-4d25-9039-00546ba0d386-sys\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.390658 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.390627 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tv2f\" (UniqueName: \"kubernetes.io/projected/be1ce9de-0f29-4d25-9039-00546ba0d386-kube-api-access-7tv2f\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.390760 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.390658 7570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be1ce9de-0f29-4d25-9039-00546ba0d386-podres\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.491325 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.491299 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be1ce9de-0f29-4d25-9039-00546ba0d386-proc\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.491448 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.491353 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be1ce9de-0f29-4d25-9039-00546ba0d386-sys\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.491448 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.491376 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tv2f\" (UniqueName: \"kubernetes.io/projected/be1ce9de-0f29-4d25-9039-00546ba0d386-kube-api-access-7tv2f\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.491448 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.491399 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be1ce9de-0f29-4d25-9039-00546ba0d386-podres\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.491448 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.491442 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be1ce9de-0f29-4d25-9039-00546ba0d386-sys\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.491647 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.491445 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be1ce9de-0f29-4d25-9039-00546ba0d386-proc\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.491647 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.491445 7570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be1ce9de-0f29-4d25-9039-00546ba0d386-lib-modules\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.491647 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.491517 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be1ce9de-0f29-4d25-9039-00546ba0d386-lib-modules\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.491647 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.491534 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be1ce9de-0f29-4d25-9039-00546ba0d386-podres\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.503890 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.503872 7570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tv2f\" (UniqueName: \"kubernetes.io/projected/be1ce9de-0f29-4d25-9039-00546ba0d386-kube-api-access-7tv2f\") pod \"perf-node-gather-daemonset-6vcj4\" (UID: \"be1ce9de-0f29-4d25-9039-00546ba0d386\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.548494 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.548473 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-t49gb_68aefc1f-0a32-4722-a494-054f46f44b5b/kube-rbac-proxy-main/0.log" Oct 31 00:21:18.549401 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.549384 7570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.575722 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.575698 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-t49gb_68aefc1f-0a32-4722-a494-054f46f44b5b/kube-rbac-proxy-self/0.log" Oct 31 00:21:18.605872 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.605846 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-t49gb_68aefc1f-0a32-4722-a494-054f46f44b5b/openshift-state-metrics/0.log" Oct 31 00:21:18.652508 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.652439 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/prometheus/0.log" Oct 31 00:21:18.680180 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.680157 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/config-reloader/0.log" Oct 31 00:21:18.705924 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.705902 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/thanos-sidecar/0.log" Oct 31 00:21:18.729587 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.729552 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/kube-rbac-proxy-web/0.log" Oct 31 00:21:18.754727 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.754710 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/kube-rbac-proxy/0.log" Oct 31 00:21:18.780541 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.780523 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/kube-rbac-proxy-thanos/0.log" Oct 31 00:21:18.807379 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.807358 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da3a65c4-b342-42d7-aad0-29e63b90125a/init-config-reloader/0.log" Oct 31 00:21:18.828886 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.828867 7570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4"] Oct 31 00:21:18.831099 ip-10-0-137-81 kubenswrapper[7570]: W1031 00:21:18.831073 7570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbe1ce9de_0f29_4d25_9039_00546ba0d386.slice/crio-191084b0e48737a786308d3cebbd7eb13590b082b9a777d20220c24a26bd8d42 WatchSource:0}: Error finding container 191084b0e48737a786308d3cebbd7eb13590b082b9a777d20220c24a26bd8d42: Status 404 returned error can't find the container with id 191084b0e48737a786308d3cebbd7eb13590b082b9a777d20220c24a26bd8d42 Oct 31 00:21:18.938130 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.938073 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-976bbf6cb-lk9st_d0b4de16-5f57-44fe-a273-a77ebd48d050/telemeter-client/0.log" Oct 31 00:21:18.944974 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.944954 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" event={"ID":"be1ce9de-0f29-4d25-9039-00546ba0d386","Type":"ContainerStarted","Data":"fa7bc76af70e64f5478d2be90d41f67cb816c34fb7f6fcf8799282db4830eeac"} Oct 31 00:21:18.945066 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.945001 7570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" event={"ID":"be1ce9de-0f29-4d25-9039-00546ba0d386","Type":"ContainerStarted","Data":"191084b0e48737a786308d3cebbd7eb13590b082b9a777d20220c24a26bd8d42"} Oct 31 00:21:18.945106 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.945089 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:18.960958 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.960942 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-976bbf6cb-lk9st_d0b4de16-5f57-44fe-a273-a77ebd48d050/reload/0.log" Oct 31 00:21:18.963958 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.963917 7570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" podStartSLOduration=0.963896285 podStartE2EDuration="963.896285ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:18.961768003 +0000 UTC m=+3690.480769093" watchObservedRunningTime="2026-10-31 00:21:18.963896285 +0000 UTC m=+3690.482897374" Oct 31 00:21:18.985238 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:18.985221 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-976bbf6cb-lk9st_d0b4de16-5f57-44fe-a273-a77ebd48d050/kube-rbac-proxy/0.log" Oct 31 00:21:19.019383 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:19.019362 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/thanos-query/0.log" Oct 31 00:21:19.043685 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:19.043662 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/kube-rbac-proxy-web/1.log" Oct 31 00:21:19.044051 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:19.044021 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/kube-rbac-proxy-web/0.log" Oct 31 00:21:19.063278 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:19.063256 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/kube-rbac-proxy/0.log" Oct 31 00:21:19.090201 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:19.090178 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/prom-label-proxy/0.log" Oct 31 00:21:19.116838 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:19.116822 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/kube-rbac-proxy-rules/0.log" Oct 31 00:21:19.144771 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:19.144749 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f65cc44c7-ljdt2_d189925d-3211-43c6-8bf0-e9aca119e00e/kube-rbac-proxy-metrics/0.log" Oct 31 00:21:23.489852 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:23.489820 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h4hmx_cb76bb5e-c0a6-4a38-8732-40193dd39cac/dns/0.log" Oct 31 00:21:23.513764 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:23.513744 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h4hmx_cb76bb5e-c0a6-4a38-8732-40193dd39cac/kube-rbac-proxy/0.log" Oct 31 00:21:23.541070 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:23.541046 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bwf5h_1ecda0b2-bd81-4e60-b31c-ef8fd2d2a22f/dns-node-resolver/0.log" Oct 31 00:21:24.223237 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:24.223208 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29890080-w2p9r_de608395-f32c-4f76-ad97-988d2feb7ca0/image-pruner/0.log" Oct 31 00:21:24.330687 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:24.330658 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vqvzf_f5f12d67-0b46-434f-8c41-a230d421eefd/node-ca/0.log" Oct 31 00:21:24.966949 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:24.966918 7570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-6vcj4" Oct 31 00:21:28.196830 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:28.196800 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dfd47b86c-gjwxh_2b7be075-e8bb-4db1-bfab-782276c29b6e/kube-auth-proxy/0.log" Oct 31 00:21:28.235456 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:28.235430 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-7rp6c_e7406dbc-b7ad-4b0c-ac37-c85b302bf76e/istio-proxy/0.log" Oct 31 00:21:28.343234 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:28.343203 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-g4dbp_229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d/istio-proxy/0.log" Oct 31 00:21:28.502251 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:28.502181 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-p5kqz_9a41f90f-265b-4a47-a2bc-9239bbd1822b/istio-proxy/0.log" Oct 31 00:21:29.411736 ip-10-0-137-81 kubenswrapper[7570]: E1031 00:21:29.411701 7570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vq6t" containerName="extractor" Oct 31 00:21:31.077648 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:31.077621 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9tcb2_284dba08-fe12-4942-a5fd-6c1e2f979a17/serve-healthcheck-canary/0.log" Oct 31 00:21:31.719763 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:31.719723 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7vq6t_d54e0641-5f0d-41bd-8b1b-900463a24cfa/kube-rbac-proxy/0.log" Oct 31 00:21:31.745626 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:31.745607 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7vq6t_d54e0641-5f0d-41bd-8b1b-900463a24cfa/exporter/0.log" Oct 31 00:21:31.772380 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:31.772355 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7vq6t_d54e0641-5f0d-41bd-8b1b-900463a24cfa/extractor/0.log" Oct 31 00:21:32.259004 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:32.258972 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dfd47b86c-gjwxh_2b7be075-e8bb-4db1-bfab-782276c29b6e/kube-auth-proxy/0.log" Oct 31 00:21:32.282558 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:32.282535 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-7rp6c_e7406dbc-b7ad-4b0c-ac37-c85b302bf76e/istio-proxy/0.log" Oct 31 00:21:32.345108 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:32.345084 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-g4dbp_229ef7e0-5e24-4f6c-b60d-e1e0bf2dc32d/istio-proxy/0.log" Oct 31 00:21:32.425944 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:32.425908 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-p5kqz_9a41f90f-265b-4a47-a2bc-9239bbd1822b/istio-proxy/0.log" Oct 31 00:21:34.517160 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:34.517130 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6986c789f7-5bhpc_d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e/manager/1.log" Oct 31 00:21:34.517874 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:34.517856 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6986c789f7-5bhpc_d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e/manager/0.log" Oct 31 00:21:34.544939 ip-10-0-137-81 kubenswrapper[7570]: I1031 00:21:34.544911 7570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6986c789f7-5bhpc_d4545d5a-09c1-4bc2-8d8f-4c89a0d5b05e/copy-manifests/0.log"